Knowledge Base tab: browse, edit, navigate & graph ~/Scout/knowledge-base/#69
Knowledge Base tab: browse, edit, navigate & graph ~/Scout/knowledge-base/#69yustme wants to merge 17 commits into
Conversation
New KnowledgeBase module mirroring the PerFileItems/Proposals conventions (service + actor writer + views): - KnowledgeBaseService: builds a recursive KBNode tree from the knowledge-base/ folder, kept in sync via FSEvents. - KnowledgeBaseFileWriter: whole-file save with a baseline-mtime conflict guard (surfaces concurrent plugin writes instead of clobbering), plus create/delete/rename. Each change is git-committed scoped to its path. - Views: two-pane browser (tree + editor) with search and "New note"; KBEditorView with Edit/Preview toggle, dirty tracking, reload, conflict alert and changed-on-disk banner; KBMarkdownPreview reading renderer. - Wired into AppState, MainWindowView (.knowledgeBase) and SidebarView. - Tests for the writer helpers, tree builder and markdown parser.
…metadata - KBMarkdownPreview renders GitHub-style tables as a grid (escaped-pipe aware), constrains prose to a reading column, and collapses the leading changelog/frontmatter into a 'History & properties' disclosure. - KBEditorView opens markdown in Preview by default; editing is one click. - Tests for table parsing, separator detection, escaped pipes, and the metadata partition.
…ch, overview - KnowledgeBaseService gains a wikilink index (stem→path, per-file out-links) rebuilt on reparse, plus resolveWikilink / outgoingLinks / backlinks / localGraph / graphStats / searchContent. - InlineMarkdownText: optional `kbWikilinkHandler` environment hook so KB wikilink clicks navigate in-app (falling back to Linear/Obsidian elsewhere). - KBRightPanel: Links/Graph toggle — outgoing links + backlinks (navigable), and a native SwiftUI Canvas force-directed local graph (pan/zoom, click to open, colored by entity type). - KBOverviewView: landing page with KB stats + quick-access tiles. - KnowledgeBaseView: three-pane layout, content search with snippets, wired wikilink navigation. - Tests: wikilink extraction, index/backlinks/local-graph/stats/search, layout.
- Add Grape (SwiftGraphs/Grape 1.x) as an SPM dependency. - Replace the hand-rolled Canvas force layout with Grape's ForceDirectedGraph (KBGraphCanvas): tap a node to navigate, drag to pan, pinch to zoom; nodes colored by entity type, labels gated by degree. - KnowledgeBaseService.fullGraph() powers a whole-KB graph on the overview. - Local graph (right panel) and global graph (overview) share KBGraphCanvas. - Tests: drop the old layout test, add a fullGraph test.
- KBTableBlockView: replace Grid (which let a long cell overflow onto following rows) with fixed-width columns + HStack rows whose height tracks the tallest cell. Per-column width derived from content, clamped and wrapped. - KBGraphCanvas: start zoomed in (initialModelTransform scale) and tune node radius/label fonts/forces so labels are readable, not microscopic. - Overview global graph: cap width and bump zoom so it isn't a tiny cluster lost in whitespace.
Switch from a View-closure annotation (rendered in fixed screen space, so labels stayed tiny when zooming) to the Text annotation overload, which Grape draws on the graph canvas and scales with the viewport transform.
Grape draws symbols/text at fixed pixel size — zoom only spreads positions, so labels stayed small. Read state.modelTransform.scale and multiply node radius and label font by it, so pinch-zoom actually enlarges nodes and text (Obsidian-like). State is Observable, so the graph re-renders on zoom.
Node drag is already supported (withGraphDragGesture pins the node under the cursor and moves it, releasing back into the simulation). Bump base node radius so the hit target is comfortable to grab with the mouse.
- KBLiveEditor: NSTextView-backed editor that styles markdown inline as you type (headings, bold/italic, code, [[wikilinks]], links, [#tags], tables, code fences; syntax markers dimmed) while keeping the text exact markdown — no round-trip, so the plugin's structured tokens are preserved. - KBEditorView: Read / Rich / Source toggle; markdown opens in Rich by default, YAML stays Source.
- KBDocSegment: parse the document into blocks with exact source line ranges, plus line-precise replaceLines / replaceCell splicers (cell pipe-escaping preserved). Only the edited block/cell is rewritten; the rest stays byte-identical for the plugin. - KBEditableView: the rendered document, editable in place — double-click a paragraph/heading/list/quote/code/table-cell to edit just that piece; single-click a [[wikilink]] still navigates. Leading frontmatter/changelog collapse into 'History & properties'. - KBEditorView: 'Read' is now this editable rendered view and the default mode; Rich (live markdown) and Source (raw) remain. - Tests for segment ranges and the splicers.
contentsOfDirectory returns symlink-resolved file URLs, but scoutDirectory kept the symlink path (~/Scout -> repos/SuperScout), so the in-KB prefix guard and repo-relative paths mismatched and every save/commit was rejected. Resolve symlinks for the KB service + writer roots and in the guard/relative-path helpers so both sides use the real path.
- PaneResizeHandle: drag the Links/Graph panel's left edge to resize (220–640pt, resize cursor on hover); width persisted via @AppStorage. - Right-panel graph rebuilds (re-centered, fresh simulation) whenever the open note changes via .id(relPath), so it always reflects the current page.
AdamVyborny
left a comment
There was a problem hiding this comment.
Knowledge Base tab — review
Solid, well-documented feature. A few things should be addressed before merge: a public-repo anonymization leak, a couple of correctness bugs (escaped-pipe wikilinks, a dropped table-cell edit), main-thread blocking in the spirit of the recent heatmap-freeze fix, and a sizeable chunk of dead/duplicated renderer code whose tests give false confidence. Details below, most-severe first.
Scope
The app: job in .github/workflows/ci.yml builds and uploads an unsigned Scout.app (relying on a manual xattr quarantine-clear). It's unrelated to the Knowledge Base feature and is obsolete if signed/notarized builds are now an option — please split it out of this PR (and prefer signing if available). The project.pbxproj changes are in scope (they add the Grape dependency the graph needs) — keep those.
Findings
1. Real company name "Groupon" in public-repo fixtures and a code comment.
Violates the repo CLAUDE.md rule "Fixtures must be anonymized … No real identifiers. Strip company/product names" (all three Scout repos are public). In ScoutTests/KnowledgeBase/KnowledgeBaseTests.swift, makeLinkedKB() writes groupon.md / # Groupon, and extractsTargetsBeforePipeDeduped / contentSearchReturnsSnippet assert on "groupon". In Scout/KnowledgeBase/KnowledgeBaseService.swift, resolveWikilink's doc comment uses e.g. `groupon`. Swap for a neutral stand-in (keep the load-bearing [[ ]] / \| structure). Lower-confidence: splitsRowHonoringEscapedPipeInWikilink uses off-list names Jan/Jordan (CLAUDE.md prescribes Alex/Priya/Sam).
2. Escaped-pipe wikilinks never resolve — KnowledgeBaseService.extractWikilinks.
The regex \[\[([^\]|]+?)(?:\|[^\]]+)?\]\] against [[people\|Alias]] captures group 1 as people\ (the \ is neither ] nor |), so stemToPath["people\"] misses. Every table-cell wikilink written in the escaped-pipe form CLAUDE.md says the KB uses → treated as dangling → no graph edge, no backlink, broken in-app navigation. The unit test only covers the unescaped [[people|Alias]], so it passes while the real form is broken.
3. Whole KB read synchronously on the main thread — KnowledgeBaseService.reparse().
KnowledgeBaseService is @MainActor; reparse() runs buildChildren (recursive dir walk + per-entry resourceValues) and buildIndex (String(contentsOf:) for every .md) on the main actor — on load and on every 250ms-debounced FSEvent (including the app's own writes). A few-hundred-note vault → hundreds of synchronous disk reads blocking the UI on every file change (same class of problem as the recently-fixed heatmap freeze). Compounding it: backlinks() re-reads files buildIndex already read, and searchContent() re-reads every file from disk on each debounced keystroke. Move parsing/indexing off the main actor and reuse the text buildIndex already loaded.
4. Large dead/duplicated renderer; the tested parser isn't the one that renders — KBMarkdownPreview.swift.
Read mode is rendered by KBEditableView / KBDocSegment. KBMarkdownPreview (the view), its Block-based parse/partition/splitFrontmatter/isMetadata pipeline, render/headingSize, and KBTableBlockView are never instantiated in production — only 4 static lexer helpers (parseHeading, parseListItem, isTableSeparator, splitRow) are live. Yet the test suite validates the dead Block parser, while the parser that actually drives rendering (KBDocSegment.segments) is largely untested. Either make KBEditableView compose KBMarkdownPreview (one renderer), or delete the dead Block pipeline + KBTableBlockView and point the tests at KBDocSegment.
5. In-place edit of a padded/ragged table cell is silently dropped — KBDocSegment.replaceCell.
The parser pads short rows for display (header 3 cols, | 4 | 5 | → ["4","5",""]). Double-clicking the empty 3rd cell and typing calls replaceCell(col: 2), which re-splits the real source line into 2 cells; guard col < cells.count (2 < 2) fails and returns source unchanged. The typed value vanishes with no error, and that cell can never be filled from the in-place editor.
6. Read-mode title sinks below "History & properties" for notes with frontmatter — KBEditableView.partition.
segments() emits frontmatter as segment 0; partition appends it to history, so when the # Title H1 arrives history.isEmpty is already false and the H1 falls into rest instead of the title slot. The metadata disclosure then renders above the title — contradicting the "substance on top" design, and diverging from the (dead) KBMarkdownPreview, which strips frontmatter first and gets it right. Hits the common case, since KB notes carry type: frontmatter.
7. Git commit failures silently swallowed — KnowledgeBaseFileWriter (save/create/delete/rename).
All four mutators do try? await gitService?.commitPaths(...). A failed commit (e.g. .git/index.lock contention while a scout-plugin session runs) leaves the file changed on disk but uncommitted, while the UI reports success — a later plugin git sync/reset can then clobber the edit or resurrect a deleted file. This defeats the guarded-write protection the actor is built around. Surface commit failures rather than discarding them.
8. Per-keystroke whole-document work on the main thread — KBLiveEditor.highlight() & KBEditableView.body.
Rich mode re-runs ~12 full-document NSRegularExpression passes (incl. unbounded [\s\S]*? frontmatter/fence patterns that backtrack badly on an unterminated fence) on a 0.12s debounce; Read mode re-parses the entire document via KBDocSegment.segments(from:) on every body evaluation. On large hub notes this stutters and risks a pathological hang. Restyle the edited/visible range, and memoize the segment parse.
Smaller / lower-confidence (worth a pass, not blockers)
KBEntityGroup.of()classifies notes by path substring (contains("issue"),research||review,/people…), ignoring thetype:frontmatter the plugin already writes.tissue-sampling.md→.issues; a renamed vault layout → everything.other. Read the ontology/frontmatter instead.KBOverviewView.quickLinkshardcodes 8 exact plugin paths; tiles silently vanish when the plugin renames/moves a hub note. Discover hubs from an index, or surface top-degree notes.- mtime-epsilon conflict guard (
0.0005s) inperformSave/detectExternalChangeis not robust on coarse-granularity filesystems (HFS+/SMB/FAT). Consider a content hash orNSFileCoordination. detectExternalChange()has noisSavingguard and runs a filesystem stat on everyservice.objectWillChange; a slow commit can transiently flash the "changed on disk" banner during the app's own save.- Duplication to consolidate: the unique-edge-set loop is copy-pasted 3× (
localGraph/graphStats/fullGraph);describe(_:)is duplicated inKBEditorViewandKnowledgeBaseViewand already drifted (the latter omitsconflict/notFound/outsideKnowledgeBase);headingSizeexists 3× with different values (18.5/16 vs 19/16.5);save()/forceSave()are near-identical;ensureInsideKBhas a dead boolean clause;KBWriterError.readFailedis never thrown.
Checked and ruled out
Non-KB surfaces are unaffected by the kbWikilinkHandler hook (defaults nil, scoped to the editor subtree). The forceSave "silent clobber" is re-guarded by performSave's own mtime recheck. The CI action pins (checkout@v6, upload-artifact@v7) are valid and already used by the existing test job. parser-corpus.json is untouched, so the checksum-sync rule doesn't apply.
|
Regarding just the build step, @jordanrburger can make it signed now. And it should be separate PR. |
Knowledge Base tab — browse, read, edit, and navigate
~/Scout/knowledge-base/Adds a full Knowledge Base section to Scout.app: a file browser + reader + in-place editor + link/graph explorer over the plugin's
knowledge-base/vault. Built to match the existing module conventions (service + actor writer + views), the editorial design system, FSEvents live-reload, and guarded + git-committed writes.Reading
[[wikilinks]].[[people\|Alias]]), with per-column widths and horizontal scroll.**Last updated:** / **Prev:** / **Parent:**changelog and YAML frontmatter fold into a closed "History & properties" disclosure so the substance is on top.Editing (three modes: Read / Rich / Source)
[[wikilink]]still navigates.[[wikilinks]],[#tags], tables, code fences; syntax markers dimmed). No round-trip, so nothing is reformatted.Navigation, search & graph
[[wikilink]]navigation (falls back to Linear/Obsidian for non-KB targets) via an environment hook onInlineMarkdownText.Shell wiring
AppStateregisters the KB service + writer;MainWindowViewadds the.knowledgeBasedetail;SidebarViewadds the row.Infrastructure
SwiftGraphs/Grape) as an SPM dependency for the graph.ci.yml): runsScoutTestsand, in a parallel job, builds and uploads an unsignedScout.appartifact so the app can be downloaded and run without a local Xcode.Tests
ScoutTests/KnowledgeBase/KnowledgeBaseTests.swiftcovers the pure logic:KBDocSegmentblock parsing with source line ranges and thereplaceLines/replaceCellsplicers (multi-line, out-of-range, pipe-escaping).Notes
~/Scout(the vault is a symlink torepos/SuperScout):contentsOfDirectoryreturns symlink-resolved file URLs, so the in-KB guard and git-relative paths now resolve symlinks on both sides.CLAUDE.md).