File: backend/app/index_data.py:208-221
The "table-aware" splitter falls back to "\n| " as a separator once "\n\n\n"/"\n\n" are not found.
For any markdown table larger than CHUNK_SIZE (1500 chars, common for financial/spec tables), RecursiveCharacterTextSplitter splits between table rows.
Overlap only carries the tail of the previous chunk, so later chunks contain data rows with no header row, exactly the "split mid-table, lose context" failure the fix was supposed to prevent.
Failure scenario: A 40-row financial table exceeding 1500 chars. Rows past the first chunk lose the column headers, so the LLM cannot map values to the correct metric/year.
See docs/rag-audit-findings.md (item 2).
File:
backend/app/index_data.py:208-221The "table-aware" splitter falls back to
"\n| "as a separator once"\n\n\n"/"\n\n"are not found.For any markdown table larger than
CHUNK_SIZE(1500 chars, common for financial/spec tables),RecursiveCharacterTextSplittersplits between table rows.Overlap only carries the tail of the previous chunk, so later chunks contain data rows with no header row, exactly the "split mid-table, lose context" failure the fix was supposed to prevent.
Failure scenario: A 40-row financial table exceeding 1500 chars. Rows past the first chunk lose the column headers, so the LLM cannot map values to the correct metric/year.
See
docs/rag-audit-findings.md(item 2).