Skip to content

Knowledge Base tab: browse, edit, navigate & graph ~/Scout/knowledge-base/#69

Open
yustme wants to merge 17 commits into
mainfrom
feat/knowledge-base
Open

Knowledge Base tab: browse, edit, navigate & graph ~/Scout/knowledge-base/#69
yustme wants to merge 17 commits into
mainfrom
feat/knowledge-base

Conversation

@yustme

@yustme yustme commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Knowledge Base tab — browse, read, edit, and navigate ~/Scout/knowledge-base/

Adds a full Knowledge Base section to Scout.app: a file browser + reader + in-place editor + link/graph explorer over the plugin's knowledge-base/ vault. Built to match the existing module conventions (service + actor writer + views), the editorial design system, FSEvents live-reload, and guarded + git-committed writes.

Reading

  • Rendered Markdown preview — headings, lists, blockquotes, code, inline formatting and [[wikilinks]].
  • GitHub-style tables rendered as real grids (escaped-pipe aware, e.g. [[people\|Alias]]), with per-column widths and horizontal scroll.
  • Reading column width-capped for legibility.
  • Noise collapse — the leading **Last updated:** / **Prev:** / **Parent:** changelog and YAML frontmatter fold into a closed "History & properties" disclosure so the substance is on top.

Editing (three modes: Read / Rich / Source)

  • Read (default) — the rendered document, editable in place: double-click a paragraph, heading, list item, quote, code block or table cell to edit just that piece. Each edit rewrites only that block's exact source range (or one cell), so the rest of the file — and the plugin's structured tokens — stays byte-identical. Single-click a [[wikilink]] still navigates.
  • Rich — a native live-markdown editor (NSTextView): the text stays exact markdown but is styled inline as you type (headings, bold/italic, code, links, [[wikilinks]], [#tags], tables, code fences; syntax markers dimmed). No round-trip, so nothing is reformatted.
  • Source — raw markdown.
  • Safe writes — atomic write + scoped git commit via a baseline-mtime conflict guard (a concurrent plugin write is surfaced, not clobbered), plus create / rename / delete with confirmation.

Navigation, search & graph

  • In-app [[wikilink]] navigation (falls back to Linear/Obsidian for non-KB targets) via an environment hook on InlineMarkdownText.
  • Links panel — outgoing links + backlinks for the open note, both navigable.
  • Local graph of the open note's neighbourhood and a global KB graph on the overview, rendered with Grape (native d3-force / SwiftUI). Tap a node to navigate, drag a node to reposition, drag the canvas to pan, pinch to zoom (node/label size scales with zoom). The Links/Graph panel is resizable (drag its edge; width persisted) and the graph is always scoped to the currently open note.
  • Full-text search across note names and contents with snippets.
  • Overview landing page — note/connection counts and quick-access tiles to the canonical hub notes.

Shell wiring

AppState registers the KB service + writer; MainWindowView adds the .knowledgeBase detail; SidebarView adds the row.

Infrastructure

  • Adds Grape (SwiftGraphs/Grape) as an SPM dependency for the graph.
  • CI (ci.yml): runs ScoutTests and, in a parallel job, builds and uploads an unsigned Scout.app artifact so the app can be downloaded and run without a local Xcode.

Tests

ScoutTests/KnowledgeBase/KnowledgeBaseTests.swift covers the pure logic:

  • file writer helpers (name normalization, repo-relative paths incl. symlink-resolved roots);
  • tree builder (sorting, pruning, relative paths, flattening);
  • markdown preview parser (headings, lists, rules, tables, separator detection, escaped pipes, metadata partition);
  • wikilink index, backlinks, local graph, full graph, stats and content search;
  • KBDocSegment block parsing with source line ranges and the replaceLines / replaceCell splicers (multi-line, out-of-range, pipe-escaping).

Notes

  • Fixes a save failure under a symlinked ~/Scout (the vault is a symlink to repos/SuperScout): contentsOfDirectory returns symlink-resolved file URLs, so the in-KB guard and git-relative paths now resolve symlinks on both sides.
  • No real identifiers in fixtures (per CLAUDE.md).

yustme added 17 commits June 29, 2026 13:37
New KnowledgeBase module mirroring the PerFileItems/Proposals conventions
(service + actor writer + views):

- KnowledgeBaseService: builds a recursive KBNode tree from the
  knowledge-base/ folder, kept in sync via FSEvents.
- KnowledgeBaseFileWriter: whole-file save with a baseline-mtime conflict
  guard (surfaces concurrent plugin writes instead of clobbering), plus
  create/delete/rename. Each change is git-committed scoped to its path.
- Views: two-pane browser (tree + editor) with search and "New note";
  KBEditorView with Edit/Preview toggle, dirty tracking, reload, conflict
  alert and changed-on-disk banner; KBMarkdownPreview reading renderer.
- Wired into AppState, MainWindowView (.knowledgeBase) and SidebarView.
- Tests for the writer helpers, tree builder and markdown parser.
…metadata

- KBMarkdownPreview renders GitHub-style tables as a grid (escaped-pipe
  aware), constrains prose to a reading column, and collapses the leading
  changelog/frontmatter into a 'History & properties' disclosure.
- KBEditorView opens markdown in Preview by default; editing is one click.
- Tests for table parsing, separator detection, escaped pipes, and the
  metadata partition.
…ch, overview

- KnowledgeBaseService gains a wikilink index (stem→path, per-file out-links)
  rebuilt on reparse, plus resolveWikilink / outgoingLinks / backlinks /
  localGraph / graphStats / searchContent.
- InlineMarkdownText: optional `kbWikilinkHandler` environment hook so KB
  wikilink clicks navigate in-app (falling back to Linear/Obsidian elsewhere).
- KBRightPanel: Links/Graph toggle — outgoing links + backlinks (navigable),
  and a native SwiftUI Canvas force-directed local graph (pan/zoom, click to
  open, colored by entity type).
- KBOverviewView: landing page with KB stats + quick-access tiles.
- KnowledgeBaseView: three-pane layout, content search with snippets, wired
  wikilink navigation.
- Tests: wikilink extraction, index/backlinks/local-graph/stats/search, layout.
- Add Grape (SwiftGraphs/Grape 1.x) as an SPM dependency.
- Replace the hand-rolled Canvas force layout with Grape's ForceDirectedGraph
  (KBGraphCanvas): tap a node to navigate, drag to pan, pinch to zoom; nodes
  colored by entity type, labels gated by degree.
- KnowledgeBaseService.fullGraph() powers a whole-KB graph on the overview.
- Local graph (right panel) and global graph (overview) share KBGraphCanvas.
- Tests: drop the old layout test, add a fullGraph test.
- KBTableBlockView: replace Grid (which let a long cell overflow onto
  following rows) with fixed-width columns + HStack rows whose height tracks
  the tallest cell. Per-column width derived from content, clamped and wrapped.
- KBGraphCanvas: start zoomed in (initialModelTransform scale) and tune node
  radius/label fonts/forces so labels are readable, not microscopic.
- Overview global graph: cap width and bump zoom so it isn't a tiny cluster
  lost in whitespace.
Switch from a View-closure annotation (rendered in fixed screen space, so
labels stayed tiny when zooming) to the Text annotation overload, which Grape
draws on the graph canvas and scales with the viewport transform.
Grape draws symbols/text at fixed pixel size — zoom only spreads positions,
so labels stayed small. Read state.modelTransform.scale and multiply node
radius and label font by it, so pinch-zoom actually enlarges nodes and text
(Obsidian-like). State is Observable, so the graph re-renders on zoom.
Node drag is already supported (withGraphDragGesture pins the node under the
cursor and moves it, releasing back into the simulation). Bump base node
radius so the hit target is comfortable to grab with the mouse.
- KBLiveEditor: NSTextView-backed editor that styles markdown inline as you
  type (headings, bold/italic, code, [[wikilinks]], links, [#tags], tables,
  code fences; syntax markers dimmed) while keeping the text exact markdown —
  no round-trip, so the plugin's structured tokens are preserved.
- KBEditorView: Read / Rich / Source toggle; markdown opens in Rich by default,
  YAML stays Source.
- KBDocSegment: parse the document into blocks with exact source line ranges,
  plus line-precise replaceLines / replaceCell splicers (cell pipe-escaping
  preserved). Only the edited block/cell is rewritten; the rest stays
  byte-identical for the plugin.
- KBEditableView: the rendered document, editable in place — double-click a
  paragraph/heading/list/quote/code/table-cell to edit just that piece;
  single-click a [[wikilink]] still navigates. Leading frontmatter/changelog
  collapse into 'History & properties'.
- KBEditorView: 'Read' is now this editable rendered view and the default mode;
  Rich (live markdown) and Source (raw) remain.
- Tests for segment ranges and the splicers.
contentsOfDirectory returns symlink-resolved file URLs, but scoutDirectory kept
the symlink path (~/Scout -> repos/SuperScout), so the in-KB prefix guard and
repo-relative paths mismatched and every save/commit was rejected. Resolve
symlinks for the KB service + writer roots and in the guard/relative-path
helpers so both sides use the real path.
- PaneResizeHandle: drag the Links/Graph panel's left edge to resize (220–640pt,
  resize cursor on hover); width persisted via @AppStorage.
- Right-panel graph rebuilds (re-centered, fresh simulation) whenever the open
  note changes via .id(relPath), so it always reflects the current page.
@yustme yustme changed the title Add Knowledge Base tab (browse + edit ~/Scout/knowledge-base/) Knowledge Base tab: browse, edit, navigate & graph ~/Scout/knowledge-base/ Jun 29, 2026
@yustme yustme requested a review from jordanrburger June 29, 2026 20:11

@AdamVyborny AdamVyborny left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Knowledge Base tab — review

Solid, well-documented feature. A few things should be addressed before merge: a public-repo anonymization leak, a couple of correctness bugs (escaped-pipe wikilinks, a dropped table-cell edit), main-thread blocking in the spirit of the recent heatmap-freeze fix, and a sizeable chunk of dead/duplicated renderer code whose tests give false confidence. Details below, most-severe first.

Scope

The app: job in .github/workflows/ci.yml builds and uploads an unsigned Scout.app (relying on a manual xattr quarantine-clear). It's unrelated to the Knowledge Base feature and is obsolete if signed/notarized builds are now an option — please split it out of this PR (and prefer signing if available). The project.pbxproj changes are in scope (they add the Grape dependency the graph needs) — keep those.


Findings

1. Real company name "Groupon" in public-repo fixtures and a code comment.
Violates the repo CLAUDE.md rule "Fixtures must be anonymized … No real identifiers. Strip company/product names" (all three Scout repos are public). In ScoutTests/KnowledgeBase/KnowledgeBaseTests.swift, makeLinkedKB() writes groupon.md / # Groupon, and extractsTargetsBeforePipeDeduped / contentSearchReturnsSnippet assert on "groupon". In Scout/KnowledgeBase/KnowledgeBaseService.swift, resolveWikilink's doc comment uses e.g. `groupon`. Swap for a neutral stand-in (keep the load-bearing [[ ]] / \| structure). Lower-confidence: splitsRowHonoringEscapedPipeInWikilink uses off-list names Jan/Jordan (CLAUDE.md prescribes Alex/Priya/Sam).

2. Escaped-pipe wikilinks never resolve — KnowledgeBaseService.extractWikilinks.
The regex \[\[([^\]|]+?)(?:\|[^\]]+)?\]\] against [[people\|Alias]] captures group 1 as people\ (the \ is neither ] nor |), so stemToPath["people\"] misses. Every table-cell wikilink written in the escaped-pipe form CLAUDE.md says the KB uses → treated as dangling → no graph edge, no backlink, broken in-app navigation. The unit test only covers the unescaped [[people|Alias]], so it passes while the real form is broken.

3. Whole KB read synchronously on the main thread — KnowledgeBaseService.reparse().
KnowledgeBaseService is @MainActor; reparse() runs buildChildren (recursive dir walk + per-entry resourceValues) and buildIndex (String(contentsOf:) for every .md) on the main actor — on load and on every 250ms-debounced FSEvent (including the app's own writes). A few-hundred-note vault → hundreds of synchronous disk reads blocking the UI on every file change (same class of problem as the recently-fixed heatmap freeze). Compounding it: backlinks() re-reads files buildIndex already read, and searchContent() re-reads every file from disk on each debounced keystroke. Move parsing/indexing off the main actor and reuse the text buildIndex already loaded.

4. Large dead/duplicated renderer; the tested parser isn't the one that renders — KBMarkdownPreview.swift.
Read mode is rendered by KBEditableView / KBDocSegment. KBMarkdownPreview (the view), its Block-based parse/partition/splitFrontmatter/isMetadata pipeline, render/headingSize, and KBTableBlockView are never instantiated in production — only 4 static lexer helpers (parseHeading, parseListItem, isTableSeparator, splitRow) are live. Yet the test suite validates the dead Block parser, while the parser that actually drives rendering (KBDocSegment.segments) is largely untested. Either make KBEditableView compose KBMarkdownPreview (one renderer), or delete the dead Block pipeline + KBTableBlockView and point the tests at KBDocSegment.

5. In-place edit of a padded/ragged table cell is silently dropped — KBDocSegment.replaceCell.
The parser pads short rows for display (header 3 cols, | 4 | 5 |["4","5",""]). Double-clicking the empty 3rd cell and typing calls replaceCell(col: 2), which re-splits the real source line into 2 cells; guard col < cells.count (2 < 2) fails and returns source unchanged. The typed value vanishes with no error, and that cell can never be filled from the in-place editor.

6. Read-mode title sinks below "History & properties" for notes with frontmatter — KBEditableView.partition.
segments() emits frontmatter as segment 0; partition appends it to history, so when the # Title H1 arrives history.isEmpty is already false and the H1 falls into rest instead of the title slot. The metadata disclosure then renders above the title — contradicting the "substance on top" design, and diverging from the (dead) KBMarkdownPreview, which strips frontmatter first and gets it right. Hits the common case, since KB notes carry type: frontmatter.

7. Git commit failures silently swallowed — KnowledgeBaseFileWriter (save/create/delete/rename).
All four mutators do try? await gitService?.commitPaths(...). A failed commit (e.g. .git/index.lock contention while a scout-plugin session runs) leaves the file changed on disk but uncommitted, while the UI reports success — a later plugin git sync/reset can then clobber the edit or resurrect a deleted file. This defeats the guarded-write protection the actor is built around. Surface commit failures rather than discarding them.

8. Per-keystroke whole-document work on the main thread — KBLiveEditor.highlight() & KBEditableView.body.
Rich mode re-runs ~12 full-document NSRegularExpression passes (incl. unbounded [\s\S]*? frontmatter/fence patterns that backtrack badly on an unterminated fence) on a 0.12s debounce; Read mode re-parses the entire document via KBDocSegment.segments(from:) on every body evaluation. On large hub notes this stutters and risks a pathological hang. Restyle the edited/visible range, and memoize the segment parse.


Smaller / lower-confidence (worth a pass, not blockers)

  • KBEntityGroup.of() classifies notes by path substring (contains("issue"), research||review, /people…), ignoring the type: frontmatter the plugin already writes. tissue-sampling.md.issues; a renamed vault layout → everything .other. Read the ontology/frontmatter instead.
  • KBOverviewView.quickLinks hardcodes 8 exact plugin paths; tiles silently vanish when the plugin renames/moves a hub note. Discover hubs from an index, or surface top-degree notes.
  • mtime-epsilon conflict guard (0.0005s) in performSave/detectExternalChange is not robust on coarse-granularity filesystems (HFS+/SMB/FAT). Consider a content hash or NSFileCoordination.
  • detectExternalChange() has no isSaving guard and runs a filesystem stat on every service.objectWillChange; a slow commit can transiently flash the "changed on disk" banner during the app's own save.
  • Duplication to consolidate: the unique-edge-set loop is copy-pasted 3× (localGraph/graphStats/fullGraph); describe(_:) is duplicated in KBEditorView and KnowledgeBaseView and already drifted (the latter omits conflict/notFound/outsideKnowledgeBase); headingSize exists 3× with different values (18.5/16 vs 19/16.5); save()/forceSave() are near-identical; ensureInsideKB has a dead boolean clause; KBWriterError.readFailed is never thrown.

Checked and ruled out

Non-KB surfaces are unaffected by the kbWikilinkHandler hook (defaults nil, scoped to the editor subtree). The forceSave "silent clobber" is re-guarded by performSave's own mtime recheck. The CI action pins (checkout@v6, upload-artifact@v7) are valid and already used by the existing test job. parser-corpus.json is untouched, so the checksum-sync rule doesn't apply.

@AdamVyborny

Copy link
Copy Markdown
Collaborator

Regarding just the build step, @jordanrburger can make it signed now. And it should be separate PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants