Skip to content

[codex] Improve XHS media text pipeline#10

Draft
jian45154 wants to merge 1 commit into
mainfrom
codex/xhs-media-pipeline-ocr
Draft

[codex] Improve XHS media text pipeline#10
jian45154 wants to merge 1 commit into
mainfrom
codex/xhs-media-pipeline-ocr

Conversation

@jian45154

Copy link
Copy Markdown
Owner

Summary

  • Extract Xiaohongshu subtitle tracks, transcripts, carousel images, shortlink note IDs, and canonical URLs more reliably.
  • Wire video transcripts into media text ingestion/reindexing and improve media pipeline status accounting.
  • Make the Tesseract OCR preset usable on Windows, validate tessdata languages, handle empty OCR as no_text, and preprocess animated GIFs through ffmpeg first frames.
  • Add regression coverage for Xiaohongshu shortlinks, subtitles, carousel media, OCR presets, no_text, GIF preprocessing, and media pipeline blockers.

Why

The live URL batches exposed missing pieces in the media pipeline: video transcripts were not fully usable as media text, image OCR was blocked by runtime discovery/language data, empty OCR output kept items stuck in not_processed, and animated GIFs failed Tesseract directly.

Validation

  • python -m pytest -> 249 passed
  • git diff --check
  • Live shortlink batch: 5/5 succeeded, media pipeline ready
  • Mixed WeChat/XHS batch: media pipeline ready after OCR/no_text/GIF handling

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant