feat: block-based markdown renderer for doc get by zjsng · Pull Request #26 · yjwong/lark-cli

zjsng · 2026-03-21T16:06:13Z

Problem

The doc get command relies on Lark's server-rendered markdown endpoint (/docs/v1/content), which has concrete limitations for AI-agent consumption: tables come back as HTML <table> tags, Mermaid diagrams are silently omitted, content is over-escaped, and we have no control over formatting. On a real production document (~500 blocks), the output is ~17K tokens — unnecessarily large for agent context windows.

Solution

Replace the server-rendered markdown with a local block-based renderer using the Docx block API (/docx/v1/documents/:id/blocks). The renderer produces clean, token-efficient markdown optimized for agent consumption — 24x smaller than raw block JSON and 3x smaller than the legacy endpoint.

The implementation covers all 42 documented block types (up from 14), reviewed against the full Lark Server API documentation. A --raw flag preserves access to the legacy server-rendered markdown.

# New default (block-based renderer)
lark doc get <id>

# Legacy behavior (server-rendered markdown)
lark doc get <id> --raw

What's included

Block-based renderer (`internal/docrender/`)

All 42 block types — text, headings, lists, code, quotes, callouts, tables, grids, images, files, iframes, Mermaid diagrams, equations, tasks, Jira issues, wiki catalogs, OKR objectives/key results, and opaque placeholders for non-renderable types (Bitable, Diagram, etc.)
Embedded sheet rendering — sheet blocks (type 30) are resolved via the Sheets API and rendered as inline markdown tables with a 200-row cap. Batch-fetches sheets sharing the same spreadsheet token, with per-sheet fallback. Uses valueRenderOption=ToString for clean text output.
Recursive table cell rendering — nested block structures inside table cells (e.g., ordered lists with sub-items, grids, file blocks) are walked depth-first instead of dropping grandchildren. Non-text leaf blocks emit placeholders mirroring renderBlock.
Blank line spacing — ensureBlankLine applied centrally in renderChildren before all standalone blocks (headings, code, dividers, tables, quotes, callouts, placeholders) via isStandaloneBlock classifier.
Markdown escaping — inline (\*_[]~) and block-level (# > - + N. ---) to prevent downstream parsers from misinterpreting document text as formatting. Sheet cell content also escaped.
Complete code language map — all 75 Lark language IDs mapped to markdown fence labels (was 18, with a Swift/Scheme bug fix)
Inline code safety — inlineCodeWrap sizes backtick delimiters based on longest run in content per CommonMark spec
URL decoding — uses url.PathUnescape (not QueryUnescape) to preserve + in URLs. Applied to links, document mentions, and iframe embeds.
Table header semantics — respects header_row property. When false, emits an empty header row so data rows aren't promoted to header. Documents header_column and merge_info as known limitations.
OKR rendering — objective (type 37) and key result (type 38) text content extracted and rendered with [objective]/[key result] prefixes. OKR container (36) renders children. Progress (39) shows [okr progress].
4 TextElement types — Equation ( $math$ ), Reminder (UTC ISO date), InlineFile, InlineBlock. Unknown elements emit stderr warning.
Style fixes — whitespace moved outside **bold** markers, adjacent markers collapsed, dynamic fence length for code-in-code
@mention resolution via batch contact API with graceful fallback to @user_id

Embedded sheet resolution (`internal/cmd/doc.go`)

Pre-resolution pattern matching @mention enrichment: extract tokens -> batch fetch -> pass via RenderOptions
parseSheetToken and mapBatchResults helpers with full test coverage
GetSheetDataAsText and BatchGetSheetDataAsText API methods with valueRenderOption=ToString
Best-effort with stderr warnings on failure; renderer falls back to [sheet: token]

New `doc images` command

Batch downloads all images from a document with 5 QPS rate limiting:

lark doc images <document_id> -o /tmp/images/

Breaking changes

doc get output has changed. Tables are now markdown (not HTML), Mermaid diagrams appear as fenced code blocks, embedded sheets render as tables (not [sheet]), embedded content shows [image: token] / [file: name] / [task: id] placeholders instead of being silently dropped, and @mentions resolve to display names. Use --raw to get the previous behavior.

Token efficiency

Measured on a production document with 38 images, 12 tables, and ~500 blocks:

Format	Output Size	Estimated Tokens	vs Blocks
Block-based markdown (`doc get`)	22 KB	~5.5K	24x smaller
Legacy markdown (`doc get --raw`)	67 KB	~17K	8x smaller
Raw block JSON (`doc blocks`)	523 KB	~130K	baseline

Test plan

109 unit tests covering all block types, escaping, styles, table ordering, nested cells, sheet rendering, spacing, inline code backticks, URL decoding, OKR content, unknown elements
Tested against production document (~500 blocks, 38 images, 12 tables, 6 embedded sheets)
--raw flag produces legacy server-rendered markdown
@mentions fall back gracefully when Contacts scope unavailable
Embedded sheets fall back to [sheet: token] when Sheets scope unavailable
Architectural review by Principal SWE — all 5 blocking findings addressed

Replace the old /docs/v1/content?content_type=markdown endpoint with a local block-to-markdown renderer using the structured Docx block API. This gives full control over output format and fixes several issues: - Tables now render as proper markdown instead of HTML <table> tags - Mermaid diagrams (AddOns blocks) render as fenced code blocks (previously invisible in the old API output) - Text formatting preserved: bold, italic, strikethrough, inline code, links - Token-efficient output for AI agent consumers (~20% smaller for tables) The old server-rendered markdown is available via --raw flag on doc get. Adds internal/docrender package with 19 unit tests covering all supported block types including edge cases for nested lists, table cell escaping, and unknown block type handling.

Subsumes PR yjwong#25 (add-ons block support). The types.go changes from PR yjwong#25 are already included in the renderer commit. This adds the documentation updates for block type 40 (Add-Ons/Mermaid).

Add RenderOptions with UserNames map to the docrender package. The doc get command now extracts unique user IDs from blocks and resolves them to display names via the contacts API before rendering. Falls back gracefully to @user_id if resolution fails (e.g., missing permissions or user not visible to the tenant app).

Replace per-user GetUser calls with single BatchGetUsers call (GET /contact/v3/users/batch) for resolving mention display names. Supports both user and tenant access tokens. Note: requires the app's contact scope range to include the mentioned users (configured in Lark Admin Console).

Print a warning to stderr with a hint about the required scope when the batch contact API fails, instead of silently falling back to user IDs.

The batch contact API returns success with empty items (instead of an error) when the app lacks contact scope visibility. Now detects this case and warns users to check their scope configuration.

Reviewed against Lark Server API documentation (docx-v1/data-structure/block.md). Renderer improvements: - Fix Swift/Scheme language ID bug (58=Scheme, 61=Swift) - Complete code language map: 18 → 75 languages - Add markdown escaping (inline: \*_`[]~ and block-level: # > - + N. ---) - Preserve URLs in plain text with <angle bracket> autolinks - Fix bold/italic whitespace: move leading/trailing spaces outside style markers - Collapse adjacent style markers (e.g., ****text**** → **text**) - Replace newlines in table cells with spaces (fixes broken table rows) - Decode URL-encoded links from Lark API New TextElement types: - Equation ($content$), Reminder (UTC ISO date), InlineFile, InlineBlock New block renderers (types 18-42): - Container blocks: Grid, GridColumn, QuoteContainer (render children) - Content blocks: File ([file: name]), Iframe ([embed: url]) - Identifier placeholders: Task, JiraIssue, WikiCatalog (with IDs) - Simple placeholders: Bitable, ChatCard, Diagram, ISV, Mindnote, Sheet, OKR Table improvements: - Prefer Table.Cells for cell ordering (canonical per API docs) - Fall back to node.children when Table.Cells is empty New command: - `doc images <document_id> -o <dir>` — batch download all images from a document New API method: - BatchGetMediaTempDownloadURLs for batch image URL resolution Test coverage: 45 tests (26 new) covering escaping, new elements, new blocks, language map completeness, table cell ordering, and existing coverage gaps. Documentation updated: CLAUDE.md, README.md, USAGE.md, skills/documents/SKILL.md

- Throttle media temp-URL requests to 5 QPS (200ms interval) to respect API rate limits and prevent partial results for image-heavy documents - Clamp H7-H9 headings to ###### (level 6) since CommonMark only supports ATX headings up to 6 levels - Use dynamic fence length for code blocks: if content contains ```, the fence uses more backticks than the longest run found - Fix JSON schema consistency in `doc images`: empty documents now return `[]imageResult{}` instead of `[]string{}` so the `images` field type is stable regardless of content - Fix Swift language ID in help text: 58 → 61 (58 is Scheme)

Embedded spreadsheet blocks (block_type 30) previously rendered as placeholder [sheet] text. The Lark API returns a sheet token for these blocks, which can be used to fetch cell data via the Sheets API. - Add SheetBlock type and Sheet field to DocumentBlock for deserialization - Add GetSheetDataAsText and BatchGetSheetDataAsText API methods with valueRenderOption=ToString for plain text cell values - Add ExtractSheetTokens and resolveSheetData following the existing mention/image pre-resolution pattern - Batch-fetch sheets sharing the same spreadsheet token, with per-sheet fallback when the batch API omits ranges - Render sheet data as markdown tables with 200-row inline cap and truncation marker - Apply markdown escaping to cell content for safety - Distinguish resolved-but-empty sheets from unresolved ones - Handle rich text segments, embedded images, and mixed cell types

renderTableCell previously only iterated direct children, dropping any nested content (e.g., ordered lists with sub-items). Replace the flat loop with a recursive collectCellText helper that walks the block tree depth-first. The helper handles three categories: - Text blocks: render inline text (text, bullet, ordered, headings) - Non-text leaf blocks: emit placeholders (file, image, iframe, sheet, task, jira, wiki) mirroring renderBlock - Container blocks: recurse into children (callout, grid, grid column, view, quote container)

Block spacing was handled per-block (each case adding trailing \n\n), but nothing ensured a blank line *before* standalone blocks when they followed compact content like text or list items. This caused headings, dividers, tables, and other blocks to run directly after preceding content without the required blank line separator. - Add ensureBlankLine helper and isStandaloneBlock classifier - Apply spacing rule once in renderChildren, covering all standalone block types uniformly (headings, code, quote, callout, divider, table, and all placeholder blocks)

Five correctness fixes identified during comprehensive review: 1. URL decoding: replace url.QueryUnescape with url.PathUnescape to preserve '+' as literal in URLs. Apply decoding to iframe URLs. 2. Inline code: add inlineCodeWrap helper that sizes backtick delimiters based on longest run in content (CommonMark spec). 3. Table headers: when header_row=false, emit empty header row before separator so data rows aren't promoted to header semantics. 4. OKR blocks: type 36 renders as container, 37/38 render objective and key result text content, 39 shows [okr progress]. Add fields to allTextBlocks and getAnyTextBlock for mention extraction. 5. Unknown text elements: emit stderr warning instead of silently dropping content.

yjwong

Testing against production documents

Tested PR against 5 real documents (mix of docx and wiki→docx, varying complexity) using both the main and PR branch builds.

Blocker: `expire_time` type mismatch — 2 of 5 docs crash

Documents containing InlineReminder elements fail with:

json: cannot unmarshal string into Go struct field
InlineReminder.data.items.text.elements.reminder.expire_time of type int64

Root cause: ExpireTime is defined as int64 in InlineReminder, but the Lark Docx API returns it as a string. The block data structure reference misleadingly lists the type as int, but the query-all-blocks endpoint documentation consistently lists expire_time as string.

Fix (3 lines):

// internal/api/types.go — change int64 → string
ExpireTime       string            `json:"expire_time,omitempty"`
NotifyTime       string            `json:"notify_time,omitempty"`

// internal/docrender/render.go — parse string to int64
expireMs, _ := strconv.ParseInt(elem.Reminder.ExpireTime, 10, 64)
t := time.UnixMilli(expireMs).UTC()

Also add "strconv" to imports and update the two reminder test cases to use string literals. I've verified this fix locally — all 5 docs render successfully after the change, and all 109 tests pass.

Content comparison results (after applying the fix)

What the new renderer captures that main misses

Significant improvement across the board:

Images: 30–40 [image: token] references per doc (main drops all images silently)
Tables: Large data tables are captured as markdown where main drops them entirely
Embeds: [jira: ...], [task: ...], [chat], [block: ...] placeholders for embedded content (main drops silently)
File attachments: Semantic [file: name] references (main renders as ambiguous escaped text)
Cleaner markdown: No over-escaping, proper markdown tables instead of raw HTML

Content losses in the new renderer

Issue	Severity	Details
Callout blocks (type 49) dropped	High	Some callouts are missing entirely — including substantive content like strategic recommendations and action items
Nested tables lost	High	Tables-inside-table-cells are flattened/dropped (e.g. data grids, role-module matrices)
Truncated multi-level blockquotes	Medium	`> >` nested blockquote content cut off in one doc
Task checkboxes → opaque UUIDs	Medium	`[task: UUID]` loses the human-readable text that main renders as `* [x] ~~task text~~`
Jira inline cards lose summary	Low	`[jira: KEY]` vs full `[Jira Key: KEY Jira Summary: ...]` in main
Block type 48 (synced blocks)	Low	Unsupported in both renderers — not a regression
@mentions as raw IDs	Expected	Graceful fallback when Contacts scope unavailable
List numbering	Low	Some numbered lists restart at `1.` instead of continuing

Size comparison (5 docs, after fix)

Output size ranged from 7–37% smaller than main on most docs. One complex doc with many markdown tables was roughly the same size. The token efficiency claims in the PR description hold up.

Verdict

The expire_time bug is a blocker — 2 of 5 production docs crash. The content gains (images, tables, embeds) are substantial and the markdown quality is much better. The callout and nested table losses are worth addressing but could be follow-up work.

The Lark Docx API returns expire_time and notify_time as JSON strings, but the struct defined them as int64, causing unmarshal crashes on documents containing InlineReminder elements (2 of 5 production docs). Introduce FlexInt64 type that accepts both JSON strings and numbers, since the API docs are inconsistent about the wire format. Add a zero-value guard in the renderer to emit [reminder] instead of [reminder: 1970-01-01] when the timestamp is missing.

…renderer # Conflicts: # CLAUDE.md # internal/api/types.go

- Nested tables: flatten nested table content into parent cells using canonical Table.Cells ordering instead of silently dropping them - Task checkboxes: resolve task details via Task v2 API and render as - [x] ~~summary~~ (completed) or - [ ] summary (incomplete), with graceful fallback to [task: UUID] when scope unavailable - Empty callouts: render placeholder with emoji ID instead of silent drop - List numbering: preserve ordered list counter across empty text blocks that Lark inserts as spacers between list items - Nested blockquotes: verified working correctly (added test coverage)

…o feat/block-based-doc-renderer # Conflicts: # internal/api/types.go # internal/cmd/doc.go # internal/docrender/render.go # internal/docrender/render_test.go

zjsng · 2026-04-03T05:56:55Z

@yjwong Thanks for the thorough production testing — this was incredibly helpful. Here's what's been addressed:

Blocker fixed

expire_time / notify_time now use a FlexInt64 custom type that accepts both JSON strings and JSON numbers, since the Lark docs are inconsistent between the block data structure reference (int) and the list-blocks endpoint (string). The renderer also guards against zero values instead of silently rendering 1970-01-01. All 5 docs should render without crashing.

Content losses addressed

Issue	Status	Details
Nested tables	Fixed	Nested tables are flattened into the parent cell using canonical `Table.Cells` ordering (cells joined with `\|`, rows with `;` )
Task checkboxes	Fixed	Task details are now resolved via the Task v2 API — renders as `- [x] ~~summary~~` or `- [ ] summary`. Best-effort: falls back to `[task: UUID]` when `task:task:read` scope is unavailable
Callout blocks	Fixed	Empty callouts now render `> [callout: emoji_id]` or `> [callout]` instead of being silently dropped. Note: type 49 doesn't exist in the API docs (callouts are type 19, which was already handled) — if you're still seeing dropped callouts, could you share the block type from the raw JSON?
Nested blockquotes	Verified working	`QuoteContainer` nesting correctly produces `> > text`. Added test coverage.
List numbering	Fixed	Ordered list counter now persists across empty text blocks (spacers Lark inserts between list items)
Jira inline cards	Known limitation	`JiraIssueBlock` only has `id` and `key` in the API — no summary field available without Jira API integration
Synced blocks (type 48)	No action	Type 48 doesn't exist in the Lark API docs (highest is 42). The default handler warns on unknown types

Other changes since last review

Merged upstream main (region-aware endpoints, table append/block update commands)
Embedded sheet blocks now render as inline markdown tables
Blank lines before standalone blocks (headings, code, tables) for cleaner markdown
Production-readiness fixes from architectural review

zjsng added 2 commits March 22, 2026 00:00

docs: add add-ons block type to documentation

83f6a48

Subsumes PR yjwong#25 (add-ons block support). The types.go changes from PR yjwong#25 are already included in the renderer commit. This adds the documentation updates for block type 40 (Add-Ons/Mermaid).

zjsng mentioned this pull request Mar 21, 2026

feat: add add-ons block support for reading Mermaid diagrams #25

Closed

zjsng added 6 commits March 22, 2026 00:16

fix: warn users when mention name resolution fails

a139836

Print a warning to stderr with a hint about the required scope when the batch contact API fails, instead of silently falling back to user IDs.

fix: warn when @mention resolution returns empty results

0cdaefc

The batch contact API returns success with empty items (instead of an error) when the app lacks contact scope visibility. Now detects this case and warns users to check their scope configuration.

zjsng force-pushed the feat/block-based-doc-renderer branch 3 times, most recently from 279d85a to f48c9d1 Compare March 24, 2026 03:57

zjsng added 4 commits March 24, 2026 11:59

zjsng force-pushed the feat/block-based-doc-renderer branch from f48c9d1 to 1c3b32f Compare March 24, 2026 03:59

yjwong reviewed Mar 27, 2026

View reviewed changes

zjsng added 4 commits April 3, 2026 12:55

Merge remote-tracking branch 'origin/main' into feat/block-based-doc-…

8104df3

…renderer # Conflicts: # CLAUDE.md # internal/api/types.go

Merge remote-tracking branch 'fork/feat/block-based-doc-renderer' int…

e36a533

…o feat/block-based-doc-renderer # Conflicts: # internal/api/types.go # internal/cmd/doc.go # internal/docrender/render.go # internal/docrender/render_test.go

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: block-based markdown renderer for doc get - #26

feat: block-based markdown renderer for doc get#26
zjsng wants to merge 16 commits into
yjwong:mainfrom
zjsng:feat/block-based-doc-renderer

zjsng commented Mar 21, 2026 •

edited

Loading

Uh oh!

yjwong left a comment

Uh oh!

zjsng commented Apr 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

zjsng commented Mar 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

What's included

Block-based renderer (internal/docrender/)

Embedded sheet resolution (internal/cmd/doc.go)

New doc images command

Breaking changes

Token efficiency

Test plan

Uh oh!

yjwong left a comment

Choose a reason for hiding this comment

Testing against production documents

Blocker: expire_time type mismatch — 2 of 5 docs crash

Content comparison results (after applying the fix)

What the new renderer captures that main misses

Content losses in the new renderer

Size comparison (5 docs, after fix)

Verdict

Uh oh!

zjsng commented Apr 3, 2026

Blocker fixed

Content losses addressed

Other changes since last review

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zjsng commented Mar 21, 2026 •

edited

Loading

Block-based renderer (`internal/docrender/`)

Embedded sheet resolution (`internal/cmd/doc.go`)

New `doc images` command

Blocker: `expire_time` type mismatch — 2 of 5 docs crash