Overview perf + caching, gray-matter fix, ferrosearch search engine - #21
Merged
Conversation
gray-matter caches the file object keyed by content BEFORE parsing, so a failed parse leaves a poisoned entry: the next parse of a byte-identical string silently returns empty data with no error. In a vault with two identical malformed notes, only the first produced a warning — the second was indexed with empty properties and leaked raw frontmatter text into tags. All frontmatter operations now route through safeMatter, which clears the cache on a parse failure so malformed content fails deterministically.
One dense fixture vault exercising every code path — weighted TF sources, heading dedup and corroboration, bigram suppression, folder-token exclusion, all noise patterns, scaffold skipping, depth limits, malformed frontmatter, and both collapse outcomes — snapshotted across three option sets. Any behavior change in the pipeline must show up as a snapshot diff.
Three behavior-preserving changes to the term pipeline: - terms are tokenized once per text; bigrams derive from the token array instead of re-running the noise-stripping regex chain - each noise regex runs only when a cheap necessary condition holds (an email needs @, a URL needs http, ...) - occurrences are counted into a small per-text map and merged into the folder accumulators by unique term, preserving first-occurrence insertion order exactly (keyword tie-breaking depends on it) Equivalence is enforced, not assumed: the pre-refactor implementation is kept verbatim as an oracle, and a differential suite fuzzes both across 30,000+ adversarial strings and 60 randomized vaults, asserting byte-identical output including Map-order-dependent tie-breaks. 3.7s -> 1.0s on a 5,000-note vault.
Single-entry cache in .napkin/overview-cache.json keyed by the whole-vault fingerprint (paths + mtimes, shared with the search cache) plus the resolved options. A hit costs one stat pass instead of reading and tokenizing every note: ~25ms vs ~1s on a 5,000-note vault. Any file add, remove, or touch invalidates; corrupt cache files are ignored and rebuilt. Tests observe caching strictly through getOverview behavior, including a frozen-mtime rewrite that must be invisible.
…gine @shift-labs/ferrosearch 0.1.1 replaces minisearch in the search core. The serialized index format is interchangeable, so existing search-cache.json blobs keep working — pinned by a migration test asserting identical files and scores from a minisearch-written cache. The cache write moves to toJsonString() (ferrosearch has no toJSON), and the index options are a single shared constant because loadJson requires the exact options the index was serialized with. Warm search path (load cached index + query) drops from ~36ms to ~14ms on a 5,000-note vault. minisearch remains a devDependency as the migration test oracle.
require("../package.json") cannot resolve inside a bun-compiled binary's
$bunfs, which broke every build:bun executable at startup. A static JSON
import bundles the version at compile time and behaves identically under
node and tsc output.
The keyword-extraction doc catches up with the pipeline it describes: frontmatter-value weighting, the full structured-noise list with the necessary-condition guards, the homogeneous-sibling collapse step, keyword corroboration filtering, and a new caching section. configuration.md gains the overview.collapse key and the auto-managed cache files. CLAUDE.md and the README stop claiming .napkin/ holds config only, and the README states the native engine's platform support.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four workstreams, each TDD'd with its contract pinned before the change:
1. gray-matter cache-poisoning fix (
7834e8d)A failed frontmatter parse left a poisoned cache entry: the next parse of byte-identical malformed content silently returned empty data — second identical broken note got no warning and leaked raw frontmatter into tags.
safeMatterevicts on failure; malformed content now fails deterministically.2. Overview performance (
6fe8bad,e5c0502,fa5b163)napkin overviewdrops to ~25ms via.napkin/overview-cache.json(mtime fingerprint + options key).3. ferrosearch swap (
72c7b77)Search now runs on @shift-labs/ferrosearch 0.1.1 — native, MiniSearch-compatible including the serialized index format. Existing
search-cache.jsonblobs keep working (migration test asserts identical files and scores). Warm search ~36ms → ~14ms. minisearch stays as the test oracle.4. build:bun fix (
35b367d)require("../package.json")broke every compiled binary at startup (pre-existing). Static JSON import fixes it; compiled binary verified standalone, native addon embeds.Plus a docs pass (
506b18b) bringing the keyword-extraction doc, configuration reference, CLAUDE.md, and README in line with reality.Verification: 388 tests / 44,957 assertions, tsc clean, biome clean, compiled binary smoke-tested.