Documentation has various problems

[dhdaines]

I'm quite impressed by the quality of table extraction from gmft! However I'm having a bit of trouble figuring out how to use it effectively because of some problems with the documentation:

The page on LLM formatting advises to use gmft._rich_text.rich_page. This seems sketchy since it appears to be an internal module, and indeed, it doesn't actually work, giving this output:

ImportError: gmft._rich_text was experimental. Functionality has been moved to gmft.formatters.page.auto.AutoPageFormatter.

But... Where is the documentation for AutoPageFormatter? I can't seem to find it anywhere on https://gmft.readthedocs.io/ and the class itself doesn't say much of anything in its docstrings.

In fact the class doesn't seem to do much of anything either. Looking at the code I think I'm actually supposed to use gmft.formatters.page.embed.embed_tables? Both the documentation and the error message above should be updated.

Also, none of the notebooks can be viewed in GitHub (maybe this is GitHub's problem...), giving this error:

There was an error rendering your Notebook: the 'state' key is missing from 'metadata.widgets'. Add 'state' to each, or remove 'metadata.widgets'.
Using nbformat v5.10.4 and nbconvert v7.16.6

[conjuncts]

Yes, you're totally correct. The documentation needs serious updating.

For these specific documentation issues, I am away and won't be able to address this issue in a timely manner, sorry. Also, sorry for responding to you late.

For the broader issue of outdated documentation, I have been contemplating gmft's outlook. There have been a proliferation of well-maintained alternatives (ie. markitdown, marker, docling, olmocr, MinerU) which specifically focus on PDF ingestion and markdown output.

In contrast, gmft must remain lightweight and focus on table extraction. This makes _rich_text awkward to incorporate.

I am close to archiving the repository temporarily until I have the manpower to revamp and clean up the repository.

[dhdaines]

For these specific documentation issues, I am away and won't be able to address this issue in a timely manner, sorry. Also, sorry for responding to you late.

No problem at all! I could perhaps find some time to update it and make a PR...

For the broader issue of outdated documentation, I have been contemplating gmft's outlook. There have been a proliferation of well-maintained alternatives (ie. markitdown, marker, docling, olmocr, MinerU) which specifically focus on PDF ingestion and markdown output.

Yes, these packages are nice but not at all lightweight - I like docling a lot but it leaks memory like a sieve 😒 - and I agree, markdown is not the right format for anything other than doing retrieval-assisted generation with an LLM. For data science obviously you want Pandas or Polars dataframes, and for preserving table structure, either OTSL or HTML is necessary.

I like HTML 😄