perf(KnowledgeHarvester): single-pass backlink index instead of per-note O(n^2) rg scan

[asdf8675309]

What

regenerateMOC and expireStaleSeedlings each counted a note's inbound
[[wikilinks]] by shelling out rg -c '[[<slug>' across the entire
KNOWLEDGE archive once per note. Regenerating the MOCs (index, promote)
or expiring seedlings (harvest) therefore runs ~N full-archive scans for N
notes — O(n^2), with one subprocess spawn each. On a real 6,419-note archive
that is 6,419 rg invocations over a 42 MB corpus, measured at ~10 minutes for
a single index.

Change

One new helper, computeBacklinks(), reads every note once and tallies each
[[target]] into a slug -> count Map. It's computed a single time per command
run and passed into regenerateMOC(domain, backlinks) and
expireStaleSeedlings(backlinks), which now do O(1) Map lookups instead of
spawning rg. Both former rg -c sites are removed. One file, no API change, no
new dependency, and it removes a child_process dependency from the hot path.

Why it's safe

The single pass is more accurate than the per-note rg, not merely equal —
and the difference is in the safe direction (it removes over-counting):

• Exact target match. The old rg -c '[[<slug>' was a prefix match, so
  [[foo]] wrongly counted toward [[foo-bar]]. The new index matches the exact
  target.
• Distinct source notes. The old code summed matching lines (rg -c); a
  note that linked a target on 3 lines counted as 3. The new index counts
  distinct source notes.
• Skips generated MOC/_*.md files and self-links. The old scan ran over the
  whole KNOWLEDGE_DIR, so each domain's auto-generated _index.md (which lists
  nearly every note as a [[link]]) inflated real note-to-note counts, and a note
  citing its own slug counted itself. The new index skips _*.md and self-links.

Both consumers tolerate the tighter count safely: "Most Referenced" in a MOC is a
cosmetic ranking, and seedling expiry only branches on > 0 references. The
practical effect is that expiry now keys off real inbound note-links rather
than MOC self-listing, which is the intended behavior. Empirically the new count
is always <= the old (see table): across the full 6,419-note corpus, 6,257
differed and every difference was the old count being higher (NEW is never
higher).

Verification

Benchmarked against the real 6,419-note archive (READ-ONLY), old approach
reproduced faithfully from the pre-change source, new approach run verbatim.

Metric	Value
Corpus (4 domains)	6,419 notes / 42 MB
OLD (per-note rg -c, full corpus)	619 s (~10 min), 96.5 ms/slug
NEW (computeBacklinks, single pass)	0.13 s warm (0.30 s cold)
Speedup	~2,000–4,900x
Count parity (OLD vs NEW), all 6,419	162 identical, 6,257 differ, all OLD-higher (NEW never higher) = the over-count corrections above (e.g. cloudflare 689 -> 5)

Tested on macOS 25.5, Bun 1.3.14, ripgrep 15.2.0. File transpiles clean
(bun build --no-bundle --target=bun, exit 0).

Scope

One file (KnowledgeHarvester.ts). No change to the CLI surface, note schema,
staging/promote flow, or any other function. computeBacklinks is a pure,
self-contained function (no I/O beyond the reads the code already did). No new
dependencies; a child_process/rg dependency is removed from the MOC path.

[danielmiessler]

This is a model PR — real-corpus benchmark, parity table, and every over-count correction checked out. Ported into source with credit, and re-verified against our own archive: counts moved only downward as you predicted, and a hand-grep of one slug matched the new index exactly. You're in the README credits.

Closing rather than merging, not rejecting: the public repo is generated from a private source tree at release, so a direct merge would be silently wiped — porting is how community work survives. Thank you.

[asdf8675309]

This is a model PR — real-corpus benchmark, parity table, and every over-count correction checked out. Ported into source with credit, and re-verified against our own archive: counts moved only downward as you predicted, and a hand-grep of one slug matched the new index exactly. You're in the README credits.

Closing rather than merging, not rejecting: the public repo is generated from a private source tree at release, so a direct merge would be silently wiped — porting is how community work survives. Thank you.

Thanks - this one was one I had fixed in v6, forgotten about, then when I ran a /Knowledge ingest and the moc rebuilding started to take several minutes, I remembered it had been resolved before and likely would be a good one that was impacting everyone.