feat(api): backfill article markdown storage - #10
Merged
Conversation
Add article-content extraction and persistence, wire it into feed ingestion, and update API docs, e2e coverage, and plans for the new markdown backfill flow.
- Propagates remaining feed-ingestion budget into article markdown fetches. - Allows existing articles with missing markdown to be retried on later runs. - Stops article enrichment when the per-run timeout budget is exhausted.
Prettier rewrote pnpm-lock.yaml to match the repository formatting rules. This unblocks the pr-quality format job and prevents the e2e gate from failing as a downstream dependency.
Keep the root format check focused on repo-owned files instead of package- manager-managed artifacts. This avoids noisy lockfile-only diffs and records the boundary decision in paired workflow learnings.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Store extracted article markdown during feed ingestion so each article
can be consumed as readable text without extra fetches and reparsing.
What Changed
contentMarkdownandcontentExtractedAttoArticleviamigration and Prisma model updates.
article-contentmodule with repository, extraction service,and dedicated controller for content operations.
Turndown, normalize markdown quality, and skip unreadable or
empty outputs.
with per-feed and per-article timeout budgets.
POST /article-content/:id/retryfor forced reprocessing and addunit/e2e coverage for extraction, timeout handling, and retry flows.
README adjustments and dependency lock updates.
Example Flow
flowchart TB A[Ingest feed] --> B[Parse and normalize items] B --> C[Persist article rows] C --> D[tryPersistArticleContent] D --> E{Markdown exists?} E -->|yes| F[Skip] E -->|no| G[Fetch article with timeout] G --> H[Readability + Turndown] H --> I[Save contentMarkdown and extractedAt]