Skip to content

feat(api): backfill article markdown storage - #10

Merged
sommio merged 5 commits into
developfrom
feat/article-markdown-backfill
Apr 17, 2026
Merged

feat(api): backfill article markdown storage#10
sommio merged 5 commits into
developfrom
feat/article-markdown-backfill

Conversation

@sommio

@sommio sommio commented Apr 17, 2026

Copy link
Copy Markdown
Owner

Summary

Store extracted article markdown during feed ingestion so each article
can be consumed as readable text without extra fetches and reparsing.

What Changed

  1. Add contentMarkdown and contentExtractedAt to Article via
    migration and Prisma model updates.
  2. Add article-content module with repository, extraction service,
    and dedicated controller for content operations.
  3. Extract readable text from article HTML with Readability and
    Turndown, normalize markdown quality, and skip unreadable or
    empty outputs.
  4. Extend feed ingestion to trigger backfill after article persistence,
    with per-feed and per-article timeout budgets.
  5. Add POST /article-content/:id/retry for forced reprocessing and add
    unit/e2e coverage for extraction, timeout handling, and retry flows.
  6. Update docs and runbooks in English and Chinese, plus API/PR
    README adjustments and dependency lock updates.

Example Flow

flowchart TB
A[Ingest feed] --> B[Parse and normalize items]
B --> C[Persist article rows]
C --> D[tryPersistArticleContent]
D --> E{Markdown exists?}
E -->|yes| F[Skip]
E -->|no| G[Fetch article with timeout]
G --> H[Readability + Turndown]
H --> I[Save contentMarkdown and extractedAt]
Loading

Compound
HARNESS

sommio added 5 commits April 17, 2026 15:03
Add article-content extraction and persistence, wire it into feed ingestion, and update API docs, e2e coverage, and plans for the new markdown backfill flow.
- Propagates remaining feed-ingestion budget into article markdown fetches.
- Allows existing articles with missing markdown to be retried on later runs.
- Stops article enrichment when the per-run timeout budget is exhausted.
Prettier rewrote pnpm-lock.yaml to match the repository formatting rules.
This unblocks the pr-quality format job and prevents the e2e gate from
failing as a downstream dependency.
Keep the root format check focused on repo-owned files instead of package-
manager-managed artifacts. This avoids noisy lockfile-only diffs and records
the boundary decision in paired workflow learnings.
@sommio
sommio merged commit 8ecf6fb into develop Apr 17, 2026
5 checks passed
@sommio
sommio deleted the feat/article-markdown-backfill branch April 17, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant