feat(settings): add max_file_size to filter oversized files

[shixi-li]

Closes #179.

Problem

settings.yml can only filter by glob, so keeping bundled or generated files out of the index means enumerating them in exclude_patterns. The issue describes a WordPress project where WooCommerce, Elementor, and LearnPress each ship minified .js bundles; there is no scalable pattern for "all of those, wherever they are, because they are large".

Change

Adds an optional max_file_size to project settings:

include_patterns:
  - "**/*.js"
exclude_patterns:
  - "**/node_modules"
max_file_size: 500KB

• Value format — a plain byte count (1048576) or a binary-unit suffix: B, KB, MB, GB, case-insensitive, fractional allowed (1.5MB). Parsing lives in parse_file_size and rejects empty, non-numeric, zero, negative, and bool values with a clear ValueError.
• Where it applies — a SizeLimitedMatcher wrapper in file_walk.build_matcher, so it composes with the existing glob and .gitignore layers and every consumer of the project's file matching picks it up from one place, rather than each walker growing its own size check. That means ccc grep skips the same files as the indexer; the module docstring already frames the matcher as the single source of truth for "which files count as part of the project", so I kept the setting on that side of the line. Happy to scope it to indexing only if you would rather grep stay size-blind.
• Default — omitting the key leaves max_file_size at None and behavior exactly as it is today.
• Boundary — the limit is inclusive, so a file of exactly max_file_size bytes is kept.

One deliberate call: a file that cannot be stat'd (broken symlink, a race with a delete) is left to the underlying matcher rather than dropped. A size limit should not be the thing that decides an unreadable file's fate, and silently dropping it would make the cap look like it had matched something it never measured.

min_file_size from the issue's "could also be useful" note is not included; it seemed better to land the requested filter first than to add a knob with no reported use case.

Tests

tests/test_file_walk.py

• oversized files are dropped while the rest of the walk is intact
• the limit is inclusive at exactly max_file_size bytes
• an unstattable file (broken symlink) survives the cap

tests/test_settings.py

• parse_file_size table over units, spacing, case, fractions, and plain bytes
• parse_file_size rejects empty / non-numeric / zero / negative / bool / None / list
• round-trip of max_file_size through save and load
• omitting the key yields None
• a hand-written max_file_size: 500KB in the YAML parses to bytes

Negative control: unwiring SizeLimitedMatcher from build_matcher turns the two size-filtering walk tests red, so they do witness the behavior.

Validation

Run against a venv without the sentence-transformers extra, since torch has no x86_64 macOS wheel on this host:

• pytest tests/test_file_walk.py tests/test_settings.py — 66 passed
• pytest tests/ -m "not docker_e2e" — 26 failed / 245 passed; the same 26 fail on an unmodified checkout in this environment (they need sentence-transformers), and the delta is +8, exactly the tests added here
• ruff check . — clean
• ruff format --check src/ tests/ — clean (README.md is reported unformatted both before and after this change, so I left it alone)
• mypy src/cocoindex_code/file_walk.py src/cocoindex_code/settings.py — clean

Docs

README's settings.yml section gains the key, the accepted formats, the inclusive-limit and default-off semantics, and a note that it applies to ccc grep too. The "after editing ..." list now mentions max_file_size, since it is a file-matching change with the same no-restart-needed workflow.

AI assistance

Written with AI assistance (Claude). The design choice worth flagging for review is the one above: putting the cap in build_matcher rather than in the indexer, which is what makes it apply to ccc grep as well.

[georgeh0]

Thanks for adding this!