[Feature]: Add an LLM-optimized export format for Hacker News comment threads

[miguelcorderocollar]

Feature Category

Extraction Quality (better output formatting)

Problem or Use Case

Problem or Use Case
The current Markdown representation of extracted Hacker News threads is readable enough for humans, but it is not ideal for LLM ingestion.

The main problems are:

• Deep nested quote chains (>, >>, >>>) are hard for models to parse reliably.
• Comment boundaries are often implicit rather than explicit.
• Parent/child relationships are visually present but not structurally clear.
• Large walls of quoted text reduce retrieval quality and increase ambiguity.
• Multiple competing ideas in one block make chunking, summarization, tagging, and thread reconstruction harder.

This matters when the goal is not just archival preservation, but downstream use by LLMs for:

• summarization
• clustering themes
• extracting arguments
• semantic search
• thread reconstruction
• converting discussions into issues, reports, or knowledge-base entries

Right now the export format preserves the original discussion shape too literally. That is good for fidelity, but bad for machine readability.

Proposed Solution

Proposed Solution
Add an optional export mode specifically optimized for LLM readability.

Recommended output formats:

1. Structured Markdown export
2. JSONL export
3. Hybrid export with both raw and normalized versions

1) Structured Markdown export

Represent each comment as an atomic record with explicit metadata.

Suggested schema:

• Comment ID
• Parent ID
• Depth
• Author (if available)
• Text
• Replies or reply IDs
• optional Summary field

Example:

Comment c001

Parent: none
Depth: 0
Author: username

Text:
Undercover mode appears to suppress AI attribution in commits and PRs.

Replies: c002, c003

This would make the thread much easier for LLMs to parse than nested blockquotes.

2) JSONL export

Provide one comment per line as structured JSON.

Example:
{"id":"c001","parent_id":null,"depth":0,"text":"..."}
{"id":"c002","parent_id":"c001","depth":1,"text":"..."}

This would be the best format for:

• embeddings
• retrieval pipelines
• classification
• deduplication
• thread graph reconstruction

3) Hybrid export

Export both:

• a raw archival Markdown version
• an LLM-optimized normalized version

That avoids the tradeoff between fidelity and usability.

Formatting rules for the normalized export

• Flatten deep quote nesting into explicit comment records
• Preserve reply relationships with IDs instead of quote indentation
• Avoid mixing multiple comments into one text block
• Keep metadata separate from content
• Preserve original order
• Optionally trim repeated quoted context in replies to reduce noise

Alternatives Considered

Alternatives Considered

Keep the current raw Markdown format

This preserves fidelity, but it is objectively worse for LLM parsing and chunking.

Use only nested Markdown headings

Better than quote pyramids, but still weaker than explicit parent/child IDs.

Post-process the raw export externally

Possible, but inefficient. This pushes a core formatting problem onto every downstream user instead of solving it once in the product.

Only offer JSON export

Good for machines, worse for humans. A hybrid approach is better.

Impact

Medium (useful for many users)

Additional Context

No response