Caching efficiency experiment for AutoIntent 0.3.1

[voorhs]

Summary

A new experiment under experiments/caching-efficiency/ that measures the efficiency of AutoIntent 0.3.1's two file-based caches (embeddings .npy cache and the LLM structured-output cache) and proposes improvements. uv-managed project; ruff + mypy --strict pass clean.

Reviewable artifacts (markdown):

• experiments/caching-efficiency/REPORT.md — findings, root causes, and propose-only solutions (small fixes → SQLite/LMDB rewrite).
• experiments/caching-efficiency/DESIGN.md — how the cache works + evaluation methodology.

Verdict

The caches are excellent for their designed workload (re-embedding the same fixed inputs across many trials): warm reads are 1000–4400× faster than recompute with negligible key overhead. Outside exact-input-match they degrade, and there are four real bugs.

Efficiency limits

• Whole-list keying → 0% reuse and 4–7× redundant compute+storage on growing datasets / k-fold CV / reordering.
• Local/fine-tuned models: get_hash() re-pickles all params every call (~57–171 ms, not memoized).
• No eviction/TTL; structured cache uses 3 inodes/entry.

Bugs found

• B1 (High): structured-output key ignores model identity → different LLMs collide (wrong cached answer served, no new call).
• B2 (Medium): embedding key omits model name; 0.3.1's offline fallback to "main" enables cross-model collisions when HF Hub is unreachable.
• B3 (Medium-High): non-atomic writes → an interrupted/concurrent write poisons an entry permanently (next read raises, no auto-recompute).
• B4 (Low/latent): the eager __init__ preload loads nothing (is_file() filter vs directory entries).

Upstream status (as of AutoIntent v0.4.0, August 2026)

The findings above are against 0.3.1. Since this experiment was conducted, three of the four bugs have been fixed upstream (verified on the dev branch, v0.4.0):

• B1 fixed in deeppavlov/AutoIntent#336 — the structured-output key now includes model_name and base_url.
• B2 fixed in deeppavlov/AutoIntent#337 — get_hash() always hashes the model name; offline path reads the SHA from the local HF cache ref file.
• B3 still open — the embedding half is addressed by the unmerged deeppavlov/AutoIntent#340 (SQLite per-utterance cache, which grew out of this experiment's §6 proposals); the structured-output cache still writes non-atomically with no torn-entry recovery.
• B4 fixed in deeppavlov/AutoIntent#331 — the preload now filters on is_dir().

REPORT.md carries the same status table at the top, so post-merge readers won't mistake the 0.3.1 bug list for the current state of AutoIntent.

How it was run

No OpenAI calls or keys: the LLM path is simulated and embeddings use a local sentence-transformers model. Cache IO is redirected to a temp dir (the real user cache is untouched). Raw outputs in results/*.{csv,json} + 3 PNGs.

cd experiments/caching-efficiency
uv sync
uv run python run_all.py && uv run python make_plots.py
uv run ruff check . && uv run mypy .

Test plan

• Skim REPORT.md (verdict + §4 bugs + §6 proposals) and DESIGN.md.
• uv sync && uv run python run_all.py reproduces results/ numbers.
• uv run ruff check . && uv run mypy . pass clean.
• Sanity-check the cross-model collision (struct_collision) and non-atomic poisoning (concurrency) findings.

🤖 Generated with Claude Code