docs: map + interconnect the .md hierarchy, AGENTS.md per folder, readability by mvalancy · Pull Request #129 · mvalancy/val-ark

mvalancy · 2026-07-23T06:46:50Z

What & why

A deep pass to make the whole .md hierarchy mapped, interconnected, and human-readable, with an AGENTS.md orienting every important folder so agents (and people) can find their way around. This project has a lot of moving parts; this gives it a navigable spine.

What's in it (45 files, docs/comments only — no code logic changed)

12 new AGENTS.md — one per key folder: docs/, docs/design/, docs/knowledge/, scripts/, scripts/lib/, scripts/services/, scripts/tools/, tests/, tests/screenshots/, web-ui/, data/, .github/. Each follows a shared convention: H1 + one-sentence purpose, a ↑-breadcrumb (·-separated) back to the repo root and doc map, a "What's here" table, "How to work here", "Gotchas" (deep-linked into gotchas.md), and "Related".
docs/README.md → canonical whole-repo doc map — start-here triage, an extended mermaid of the doc tree, and a grouped master inventory. Root README.md / AGENTS.md / CLAUDE.md / .agents/README.md are cross-linked into it.
Large files made navigable — top-of-file navigation-index comments on server.js and the big shell libs (_common.sh, librarian.sh, update.sh, download-models.sh) and web-ui/styles.css; a section→line-range map for the single-file SPA in web-ui/AGENTS.md; and prepended TOC/index tables on the two largest knowledge files (gotchas.md, decisions.md) — prepend-only, zero body edits.
Accuracy fixes (verified against code) — self-heal writes selfheal.json+heal-events.jsonl while the librarian writes health.json (these were conflated); #69 notification center marked shipped (0.1.16); moderation documented as-built (llama.cpp / Llama-Guard-3 / llama-mtmd-cli), not the aspirational ONNX-on-NPU; 3 broken HTTPS-guide links repointed to ENCRYPTION.md; tool-count drift reconciled (50 tool scripts / ~250 catalog items, drift-proof numberless heading).

Security / hygiene fix

Sanitized a pre-existing real host name (nas-5sgf) that was committed in tests/README.md — replaced with the <ark-host> placeholder. git grep confirms it's gone repo-wide. (Public repo — Prime Directive 1.)

Verification

Independent 3-lens review (links/anchors · convention+accuracy-vs-code · secrets+non-destructive) with an adversarial verify pass — 3/3 approve, 0 blocking:

Links: ~570 internal links across 39 changed .md files resolve (unit-tested slug resolver + negative tests); gotchas.md 25/25 and decisions.md 22/22 TOC self-anchors resolve.
Accuracy: all six code-backed claims confirmed at HEAD (selfheal/health json, 50 tools, /api/status/notifications, moderation runtime, both TSV schemas, TOOL_IDS/MODEL_SLUGS counts).
Safety: secrets clean; gotchas.md/decisions.md are pure prepends (43/0, 30/0 — zero deletions, bodies byte-identical); code files comment-only (node -c + bash -n pass; CSS delimiters balanced; web-ui/index.html untouched).

Known follow-up (out of scope, pre-existing)

docs/knowledge/decisions.md: the #69 entry sits at the bottom of the body (below the "Earlier project facts" catch-all) rather than newest-first; the index faithfully mirrors the body. Left as-is to preserve the prepend-only guarantee; worth a one-line body move in a later pass.
docs/TOOLS.md keeps its internally-consistent curated "(45 Tools)" catalog (pie chart sums to 45); reconciling it to the full 50-script set is a content task, not a hierarchy one.

🤖 Generated with Claude Code

Make docs/README.md the ONE canonical whole-repo doc map: grouped master inventory of every tracked .md (root governance, docs guides, docs/design, docs/knowledge, .agents, .github, and the per-folder AGENTS.md being created across this cleanup), an extended mermaid graph showing design/knowledge/.agents subtrees, a "Start here" triage table, and a keep-in-sync rule. Replace the stale "45 tools" magic number with a source-referenced figure. Cross-link the backbone: root README gets a compact TOC + a single "Documentation map" block (four hubs + governance quartet); root AGENTS.md gets a "Doc map — where everything lives" section + a definition-of-done item; CLAUDE.md up-links AGENTS.md + .agents/; .agents/README.md notes it implements workflow.md/governance.md and gains a doc-map footer; docs/knowledge/README.md gains a .agents/ routing row + footer; docs/design/README.md gains a footer and its status stamp bumps 0.1.14 → 0.1.17. Also sanitize a leaked host name in README's test example (nas-5sgf → <ark-host>) per Prime Directive 1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…x headers Add four AGENTS.md orienting the ENGINE layer and fix staleness in scripts/README.md, plus in-file navigation-index comments (docs/comments only — no executable logic changed): - scripts/AGENTS.md — engine overview (entry points, bulk mirrors) + a server.js section navigation map (line ranges for the // ==== banners, ~48 /api/… routes, /app/<id>/, /kiwix/, /sources, /bootstrap.sh). - scripts/lib/AGENTS.md — sourced/required building blocks: valark-env, curation pipeline, fail-closed moderation, auth/commission, local CA. - scripts/services/AGENTS.md — Community LAN service runners (fixed ports, /app/<id>/ proxy, fd-8 close, forum burst, arm64 rebuilds). - scripts/tools/AGENTS.md — tool-mirroring subsystem: the five script shapes + _common.sh contract; points at the /CLAUDE.md checklist. - scripts/README.md — tool count 43→50; lib/ table gains moderation.sh, mod-sweep.sh, tls.sh, auth.js, commission.js, catalog-parse.js; add the valark CLI + mirror-self.sh + a Community services section + footer links. - Header nav indexes: server.js (// ==== section map), tools/_common.sh (function index), update.sh (subcommand→function map), download-models.sh (category index), librarian.sh (dispatch index). Verified: node -c scripts/server.js; bash -n on every edited .sh; all relative doc links resolve; no host/IP/cred values (PUBLIC repo). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…oderation accuracy Documentation cleanup for the docs/ guide set, docs/design, and docs/knowledge. Accuracy fixes (verified against code): - Self-Healing Loop mermaid (ARCHITECTURE.md) + loop step 12 (LIBRARIAN.md): the loop writes state/selfheal.json + heal-events.jsonl, NOT state/health.json — health.json is the librarian's maintain report (loop.sh:45/303 vs librarian.sh:35/493). - roadmap.md + current-state.md: bump "as of" stamps 0.1.14 -> 0.1.17 and move notification center (#69 slice 1, GET /api/status/notifications) to shipped (0.1.16). - safety-moderation.md: add an "As shipped" note — moderation runs on llama.cpp (Llama-Guard-3 + tiny VLM via llama-mtmd-cli), not the design's ONNX/NPU head. - Fix 3 broken HTTPS-guide links (deployment/admin-console/access-identity) -> ../ENCRYPTION.md. Orientation + navigation: - New AGENTS.md for docs/, docs/design/, docs/knowledge/ (reference-vs-guide split, canonical-source table, route table, lockstep-tracker rule). - Standard back-link headers on PLATFORMS/SECURITY-AUDIT/ENCRYPTION/ARM64-NAS/MODEL_INVENTORY. - Anchor-linked TOCs on ARCHITECTURE/SECURITY-AUDIT/MODEL_INVENTORY/COMMUNITY/research-brief, plus a "line numbers frozen 2026-06-06" banner above the SECURITY-AUDIT finding tables. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…anup Add per-directory AGENTS.md orientation maps and improve navigability of the two big append-only knowledge logs, all doc/comment-only (no JS/CSS logic). New AGENTS.md (shared convention: H1 + breadcrumb + What's here / How to work here / Gotchas / Related, claims verified against code): - tests/ — the one test library (run-all.sh entry point, add-a-suite) - tests/screenshots — the Playwright suite (file:// vs :3001, worktree gotcha) - web-ui — zero-dep SPA navigation map (section -> line index) - data — the two catalog TSVs (schemas + consumers) - .github — CI/release automation + contributor scaffolding Edits (readability only; bodies NOT reordered): - gotchas.md — prepend a themed table-of-contents (auto-slug anchors) - decisions.md — prepend a newest-first index table mirroring body order - styles.css — prepend a section index comment (post-insertion line numbers) - web-ui/README.md — fix stale tool-count/category/endpoint claims (reference the TOOLS/TOOL_IDS source of truth, 7 categories), normalize the footer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- tests/README.md: replace real host name with <ark-host> placeholder in the two services-e2e examples (public repo — Prime Directive 1). - README.md: reconcile "What's Included (45 Tools)" heading to a numberless, drift-proof "What's Included" with an accurate lead (50+ tools, ~250 catalog items); update the Contents TOC anchor to #whats-included; fix the Project Structure comment 45 → 50 tool scripts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…right retries (#130, #131) * chore(ci): add offline doc-link + secret/host-leak guards to the test gate (#130) The #129 doc cleanup surfaced two defect classes CI never caught: a real host name (nas-5sgf) committed in tests/README.md, and no verification that the .md hierarchy's internal links/anchors resolve. Both are cheap to catch mechanically. - tests/lib/md_link_check.py + test-doc-links.sh: offline resolver for every internal Markdown link + #anchor across all tracked .md (GitHub-slug compatible; honors <a name>/id=). Fails on any broken path or dead anchor. - tests/lib/secret_scan.py + test-secrets.sh + secrets-allowlist.txt: offline scan that fails on a tracked .env/private-key, a concrete RFC1918 private IP, a private-TLD host (.local/.lan/...), or a bare single-label URL host (the nas-5sgf class). Public dotted domains, loopback, and $/{}/<> placeholders pass; reviewed exceptions live in the allowlist (a ratchet — new leaks still fail). - Both auto-discovered by run-all.sh's test-*.sh glob (gate locally + in CI) and run as a fail-fast CI step BEFORE the Playwright browser install. - Proven by negative tests: catches nas-5sgf, a private IP, an .internal FQDN, a tracked .env, and a .pem; passes github.com/localhost/redis/placeholders. - Docs: tests/AGENTS.md (+ allowlist gotcha), tests/README.md, decisions.md entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): remove re-leaked host name; harden secret + link guards (#130) Review of #131 caught that the new files re-committed the very host name #129 scrubbed (in secret_scan.py's docstring incl. the full URL form, test-secrets.sh, and the decisions.md entry) — and the scanner couldn't self-catch it (own files SKIP_EXACT'd; the prose mention had no scheme). Removed all four; git grep clean. Hardening from the same review's non-blocking findings (all zero-FP on the tree, negative-tested): - secret_scan: URL_HOST_RE now covers DB/queue/socket schemes (redis/postgres/mongodb/amqp/mysql/mqtt/ws(s)) — a real host in a connection string is the same leak class; bracketed IPv6 literals survive capture and a private ULA (fc00::/fd00::) in a URL host is flagged (public IPv6 passes). - secret_scan: prefixed *.env files (production.env) now caught, matching the docstring; example/sample/template/dist env files stay exempt. - secret_scan: high-signal inline content secrets — private-key bodies and AWS access-key ids — in any tracked text file. - md_link_check: reference-style links `[text][label]` (defined labels only, so incidental `[x][y]` prose is ignored), internal `<a href>`, and `?query` stripping so `real.md?v=1#frag` resolves. Both guards remain 0-FP across 63 .md files; decisions.md stays a pure prepend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(ci): make the Playwright gate flake-resilient — CI retries + forbidOnly (#132) The escattr-sweep catalog test intermittently failed CI (run 29987925904: .catalog-card not found in 10s) though it passes on dev and #131 touches no UI/spec code — a genuine flake from the SPA's async render timing (root cause tracked in #132: loadCatalog fires once per route with no retry on a transient no-op). Add `retries: process.env.CI ? 2 : 0` so a real regression still fails every attempt while a transient flake self-heals, and `forbidOnly` so a stray `.only` can't silently green-light a partial CI run. This is resilience, not a mask — #132 tracks the product-side fix. Config validated (387 tests list, exit 0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Review of #131 caught that the new files re-committed the very host name #129 scrubbed (in secret_scan.py's docstring incl. the full URL form, test-secrets.sh, and the decisions.md entry) — and the scanner couldn't self-catch it (own files SKIP_EXACT'd; the prose mention had no scheme). Removed all four; git grep clean. Hardening from the same review's non-blocking findings (all zero-FP on the tree, negative-tested): - secret_scan: URL_HOST_RE now covers DB/queue/socket schemes (redis/postgres/mongodb/amqp/mysql/mqtt/ws(s)) — a real host in a connection string is the same leak class; bracketed IPv6 literals survive capture and a private ULA (fc00::/fd00::) in a URL host is flagged (public IPv6 passes). - secret_scan: prefixed *.env files (production.env) now caught, matching the docstring; example/sample/template/dist env files stay exempt. - secret_scan: high-signal inline content secrets — private-key bodies and AWS access-key ids — in any tracked text file. - md_link_check: reference-style links `[text][label]` (defined labels only, so incidental `[x][y]` prose is ignored), internal `<a href>`, and `?query` stripping so `real.md?v=1#frag` resolves. Both guards remain 0-FP across 63 .md files; decisions.md stays a pure prepend. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

mvalancy and others added 9 commits July 22, 2026 23:22

merge docs/cleanup-hub into hierarchy-cleanup

e47bd64

merge docs/cleanup-scripts into hierarchy-cleanup

92cb90b

merge docs/cleanup-docs into hierarchy-cleanup

49cab3c

merge docs/cleanup-tests into hierarchy-cleanup

04588ec

mvalancy merged commit 218ffca into dev Jul 23, 2026
1 check passed

mvalancy deleted the docs/hierarchy-cleanup branch July 23, 2026 07:05

This was referenced Jul 23, 2026

CI hardening: catch doc-link rot and secret/host-name leaks in the test gate #130

Closed

chore(ci): doc-link integrity + secret/host-leak guards in the test gate (#130) #131

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: map + interconnect the .md hierarchy, AGENTS.md per folder, readability#129

docs: map + interconnect the .md hierarchy, AGENTS.md per folder, readability#129
mvalancy merged 9 commits into
devfrom
docs/hierarchy-cleanup

mvalancy commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mvalancy commented Jul 23, 2026

What & why

What's in it (45 files, docs/comments only — no code logic changed)

Security / hygiene fix

Verification

Known follow-up (out of scope, pre-existing)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant