Skip to content

Ingestion: oversized scanned PDFs fail the vision (Pagehound) extraction pipeline #37

Description

@jaysanderson

What

A 125-page / 22 MB scanned FRDC PDF (1995-167-DLD.pdf) failed ingestion. Routed to Pagehound vision (low text-density across all pages); the upload succeeded but platform processing returned ERROR - a 125-page vision job is too large to complete (it partially classified, then errored).

Recovery (manual, done)

Re-ingesting the same file with default extraction processes fine (text-only). Full OCR would need splitting into ~5x25-page chunks.

Systematic fix

Cap vision routing by page count: docs over ~60 pages go to default regardless of text density, since the vision job will not complete. NOTE: the corpus loader currently lives outside this repo (~/frdc-docs/load_corpus.py) so it may be tracked separately - filing here for visibility. If the app ever performs ingestion/extract-strategy selection, apply the same cap there.

Cleanup

The failed vision resource and the recovered default resource for 1995-167-DLD.pdf coexist in the FRDC box (same filename) - deduplicate post-load (keep PROCESSED, remove ERROR).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions