fix(prompt): start PromptCache workers lazily instead of on import

[aloktripathi1]

Closes (link the cache thread issue if you filed one separately)

Importing fi.prompt (per the README's own quickstart, from fi.prompt import Prompt, PromptTemplate, ModelConfig) eagerly spawns 2 background
_RefreshWorker daemon threads via the module-level prompt_cache singleton.

These threads poll an empty queue forever, because refresh_async(), the
only thing that feeds work to them, is never called anywhere in the SDK.
get_by_name() explicitly opts out, per its own comment: "Async refresh logic
is not applied here to simplify the fallback flow." So every user importing
fi.prompt gets 2 permanently idle threads for zero benefit.

Verified with threading.active_count() before and after the README import:

• Before fix: 2 PromptCacheWorker threads spawn immediately on current main
• After fix: 0 threads spawn on import

Fix: _TaskManager now starts its workers lazily on the first submit() call
via double-checked locking, instead of eagerly in init. No public API
change, refresh_async() behaves exactly as before, the threads just start
on demand if and when something actually uses them.

4 new regression tests:

• No workers exist immediately after construction
• Workers start correctly on the first submit() call
• Repeated _ensure_started() calls don't spawn duplicate workers
• _shutdown() is safe to call even if nothing was ever submitted

Note: the 2 failing tests in test_annotation_queues.py
(TestAnalytics::test_get_analytics, TestExport::test_export_json) are
pre-existing on main and unrelated to this change.

[NVJKKartik]

Diagnosis is right and I reproduced it. Clean 3.12 venv, uv pip install -e python/, then the README import:

origin/main   ->  ['MainThread', 'PromptCacheWorker-0', 'PromptCacheWorker-1']
this branch   ->  ['MainThread']

cache.py:94 is also the only .start() under python/fi/, so nothing else is spawning on that import path. And your note about test_annotation_queues.py checks out, both of those fail on origin/main too.

My problem is with the fix, not the finding.

1. The thing you're making lazy is dead code. Delete it instead.

_TaskManager.submit() has exactly one caller, refresh_async(). refresh_async() has zero callers. Not in python/, not in typescript/, not exported from fi/prompt/__init__.py, not in the README, not in any example. So _RefreshWorker, _TaskManager, refresh_async and _refreshing_keys are ~60 lines nothing reaches.

This PR adds 20 lines of lazy-start machinery plus 38 lines of tests so that dead code stops being harmful. Deleting it fixes the same bug with a negative diff and no lock to reason about. client.py:591 already says the SWR path was skipped on purpose. Bring the workers back when something actually calls them.

One caveat to put in the description if you go this way: it also drops max_workers from PromptCache.__init__. Same bucket though, public-named but undocumented and unused, and we're pre-1.0.

2. If you want to keep refresh_async as a deliberate hook, use ThreadPoolExecutor

It's already lazy. Threads spawn on first submit, and it registers its own atexit join:

>>> p = ThreadPoolExecutor(max_workers=2, thread_name_prefix="PromptCacheWorker")
>>> [t.name for t in threading.enumerate()]
['MainThread']
>>> p.submit(lambda: None).result()
>>> [t.name for t in threading.enumerate()]
['MainThread', 'PromptCacheWorker_0']

That removes _RefreshWorker and _TaskManager both, ~55 lines, and no hand-rolled double-checked locking. We already wrap TPE in fi/utils/executor.py.

3. This is the one I actually need changed: none of the four tests catch the bug

I ran them against origin/main. Two fail on assert tm._workers == [], two on AttributeError for _started / _ensure_started. They fail because the attributes don't exist yet, not because behavior regressed.

So if the bug comes back a different way (someone calls refresh_async at import, adds an eager thread elsewhere, starts a pool in __init__) all four still pass. And any refactor of the lazy mechanism breaks all four for no reason.

The test worth writing is the check you already did by hand. Subprocess, because in-process it's order dependent, another test importing fi.prompt first makes it vacuous:

def test_importing_fi_prompt_spawns_no_threads():
    code = "import threading, fi.prompt; print([t.name for t in threading.enumerate()])"
    out = subprocess.run([sys.executable, "-c", code],
                         capture_output=True, text=True, check=True).stdout
    assert "PromptCacheWorker" not in out

That one survives refactors and fails on any reintroduction.

Smaller stuff

4. Nothing exercises the lock. test_ensure_started_is_idempotent is three sequential calls on one thread, and the _started flag alone handles that. Delete the lock and the test still passes. If you keep the DCL, fire N threads at submit() at once and assert len(tm._workers) == num_workers.

5. time and pytest are imported in the new test file and never used. No lint in CI so nothing catches it.

6. self._workers: list = [] loses the element type. Module already has from __future__ import annotations, so list[_RefreshWorker]. _ensure_started is missing -> None.

7. _ensure_started sits under the # Public API banner but it's private.

8. Description still has the template placeholder: "Closes (link the cache thread issue if you filed one separately)".

9. Not a regression, just noting it: _shutdown() doesn't reset _started or _workers, so a submit() after shutdown queues a task nobody runs. Same on main. Goes away with 1 or 2 anyway.

One thing you didn't mention that I liked: deferring the atexit.register also stops leaking every _TaskManager we construct, since atexit holds a strong ref to the bound method.

So, 1 (or 2 if you're keeping the hook), and 3 either way.

[aloktripathi1]

@NVJKKartik went with option 1, deleted refresh_async/_RefreshWorker/_TaskManager
entirely rather than making it lazy. You were right that it's genuinely
dead code, zero callers anywhere, not in all, not referenced in
README or examples, and it lines up with what client.py already says:
"Async refresh logic is not applied here to simplify the fallback flow."
Deleting matched that existing intent rather than fighting it.

Net diff is -138/+13. get(), get_stale(), set(), invalidate(), make_key(),
and the prompt_cache singleton are untouched.

Replaced all 4 tests with the subprocess-based one you wrote, confirmed it
correctly fails against the original eager-spawn code and passes on the
fix. Also dropped max_workers/DEFAULT_REFRESH_WORKERS along with it, same
bucket as you said, unexported and unused.

68/70 tests pass locally, the 2 failures are test_annotation_queues.py's
pre-existing pydantic issues, unrelated, fail identically on main.

Thanks for the thorough review, caught a much better fix than what I
originally shipped.