diff --git a/CHANGELOG.md b/CHANGELOG.md index 9098ce3..e86dd49 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,46 @@ ## Unreleased -## 0.4.0 - 2026-05-28 +## 0.5.0 - 2026-05-27 + +PAM 0.5.0 adds a measurable file-only retrieval gate for agents that cannot use +semantic memory search. The graph schema remains `pam-graph-v1`; this release +also includes the migration-enforcement tooling introduced after 0.3.0. + +### Added + +- `memory:graph:coverage` CLI for graph-first coverage reports. +- Default file-only coverage scenario at `benchmarks/file-only-coverage.json`. +- Migration guide for 0.4.0 users adopting file-only coverage. +- Runtime guidance requiring graph-first lookup before broad corpus scans. + +### Improved + +- Graph memory now includes aliases and nodes for the coverage scenario. +- Coverage output reports aggregate read volume, targeted source counts, hit + rate, and `PASS` / `PARTIAL` / `BLOCKED` status without raw source text or + absolute paths. +- README documents the default 5-file / 100 KB graph-first budget and 80% hit + rate target. + +### Compatibility + +- `memoryFormat` remains `graph-v1`. +- `graphSchemaVersion` remains `pam-graph-v1`. +- Existing 0.4.0 graph JSONL files remain valid. + +### Added: Kimi Code CLI integration + +- `tools/kimi/install-mcp.mjs` registers the PAM MCP server with Kimi Code CLI + by writing an absolute-path entry into `~/.kimi/mcp.json`. +- `tools/kimi/templates/mcp.fragment.json` is the config template used by the + installer. +- `tools/kimi/docs/pam-kimi-layer.md` covers install, verify, uninstall, and + ad-hoc `--mcp-config-file` usage. +- `npm run kimi:mcp:install` shortcut for the installer. +- `docs/mcp-server.md` and `AGENT_BOOTSTRAP.md` gain Kimi-specific sections. + +## 0.4.0 - 2026-05-21 PAM 0.4.0 ships the optional agent layer end-to-end: a local stdio MCP server, the curator and scribe reference subagents, and a Claude Code @@ -64,18 +103,6 @@ existing 0.3.0 graph-v1 workspaces work without modification. - `tools/claude/install-statusline.mjs` (`npm run claude:statusline:install`) for users who want only the statusline without the full plugin. -### Added: Kimi Code CLI integration - -- `tools/kimi/install-mcp.mjs` registers the PAM MCP server with Kimi Code - CLI by writing an absolute-path entry into `~/.kimi/mcp.json`. -- `tools/kimi/templates/mcp.fragment.json` is the config template used by - the installer. -- `tools/kimi/docs/pam-kimi-layer.md` covers install, verify, uninstall, - and ad-hoc `--mcp-config-file` usage. -- `npm run kimi:mcp:install` shortcut for the installer. -- `docs/mcp-server.md` and `AGENT_BOOTSTRAP.md` gain Kimi-specific - sections. - ### Added: Docs and tests - `docs/mcp-server.md` (generic MCP docs), `tools/claude/docs/curator-agent.md`, @@ -88,10 +115,9 @@ existing 0.3.0 graph-v1 workspaces work without modification. ### Compatibility -The markdown + JSONL contract is unchanged; the new layer is additive -runtime. Existing 0.3.1 graph-v1 workspaces work unchanged, including agents -that read `memory/` by hand without the MCP server. See -`migrations/0.3.1-to-0.4.0-agent-layer.md` for the version-bump checklist. +No migration required. The markdown + JSONL contract is unchanged; the new +layer is additive runtime. Existing 0.3.0 graph-v1 workspaces work unchanged, +including agents that read `memory/` by hand without the MCP server. ### Privacy diff --git a/README.md b/README.md index 2e440a7..89f3fca 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,9 @@ The important pieces are: - [tools/memory-maintenance.mjs](tools/memory-maintenance.mjs): optional maintenance CLI for rotating logs, rebuilding archive indexes, and refreshing summaries. +- [benchmarks/file-only-coverage.json](benchmarks/file-only-coverage.json): + default smoke scenario for testing graph-first retrieval with file-only + agents. ## Who It Is For @@ -224,6 +227,7 @@ Commands: ```bash npm test npm run memory:graph:validate +npm run memory:graph:coverage -- --json npm run memory:graph:query -- --q PAM --json npm run memory:detect -- --json npm run benchmark:current @@ -259,6 +263,28 @@ The optional `memory:graph:*` scripts validate and query this data, but agents can also use plain text search or any JSONL parser. Node is not required to read or manually update the memory. +### File-Only Coverage + +PAM 0.5.0 adds a file-only coverage check for agents that do not have semantic +memory search. It compares the graph-first read path against a broad corpus +scan and verifies that realistic queries resolve through aliases/nodes before +opening long source files. + +```bash +npm run memory:graph:coverage -- --json +``` + +The default budget is: + +- maximum 5 graph files before source fallback; +- maximum 100 KB before source fallback; +- maximum 1 targeted source file per query; +- at least 80% `PASS` results in the scenario. + +Add project-specific queries to a JSON scenario and pass it with +`--scenario `. The coverage output is aggregate-only: it reports paths, +byte counts, statuses, and node ids, not raw source text. + ## Benchmarks The benchmark tool records aggregate public metrics only: file counts, bytes, @@ -290,6 +316,17 @@ The benchmark is a read-volume proxy, not a model billing report. It estimates the amount of text an agent needs to inspect for the same generic memory lookup path before and after graph-v1. +## OpenClaw Daily Graph Maintenance + +For OpenClaw-style workspaces, recurring PAM maintenance should keep the graph +useful as a compact knowledge map. It should validate the graph, scan recent +durable memory, promote safe source-traced records, update the catalog, and +write a concise maintenance report. + +It should not be only an installation acceptance-criteria checklist. See +[docs/openclaw-daily-graph-maintenance.md](docs/openclaw-daily-graph-maintenance.md) +for the reusable runbook. + ## Maintenance Safety Model The maintenance tool is conservative: diff --git a/benchmarks/file-only-coverage.json b/benchmarks/file-only-coverage.json new file mode 100644 index 0000000..8be162d --- /dev/null +++ b/benchmarks/file-only-coverage.json @@ -0,0 +1,26 @@ +{ + "name": "PAM file-only coverage smoke", + "description": "Queries an agent should route through graph files before opening long sources.", + "queries": [ + { + "q": "PAM", + "expectedId": "project:pam" + }, + { + "q": "runtime guide", + "expectedId": "doc:runtime" + }, + { + "q": "graph cli", + "expectedId": "tool:graph" + }, + { + "q": "markdown migration", + "expectedId": "migration:markdown-to-graph-v1" + }, + { + "q": "agent-facing runbooks", + "expectedId": "principle:agent-facing-runbooks" + } + ] +} diff --git a/docs/openclaw-daily-graph-maintenance.md b/docs/openclaw-daily-graph-maintenance.md new file mode 100644 index 0000000..ed4e81e --- /dev/null +++ b/docs/openclaw-daily-graph-maintenance.md @@ -0,0 +1,111 @@ +# OpenClaw Daily Graph Maintenance + +Use this runbook when PAM is installed in an OpenClaw-style workspace and the +runtime supports scheduled agent jobs. + +The maintenance job must keep the PAM graph useful as a compact, source-traced +knowledge map. It must not degrade into an installation acceptance checklist. +Acceptance criteria are useful during setup and audits, but daily maintenance is +about validating, scanning, and promoting safe graph updates. + +## Scope + +Default writable paths: + +```text +memory/graph/catalog.json +memory/graph/nodes.jsonl +memory/graph/edges.jsonl +memory/graph/aliases.jsonl +memory/maintenance/ +``` + +Do not rewrite OpenClaw-owned or project-owned sources by default, including: + +```text +MEMORY.md +memory/**/*.md +compiled wiki pages +AGENTS.md +project-specific memory conventions +``` + +If a useful update requires a non-PAM-owned write, report it as blocked and ask +for explicit approval. + +## Daily job flow + +1. Read local policy first, then `memory/pam.version.json`, + `memory/agent-memory/pam-runtime.md`, `memory/agent-memory/pam-openclaw.md`, + and `memory/graph/catalog.json`. +2. Validate `memory/graph/*.jsonl`. +3. Read the latest PAM maintenance report to establish the previous state. +4. Do a bounded scan of recent durable memory since the previous run, usually + the last 24-48 hours. +5. Promote only stable, useful, non-sensitive records into the graph: + projects, systems, decisions, tasks, risks, blockers, packages, + repositories, policies, and automations. +6. Upsert records with stable ids such as `project:*`, `system:*`, + `decision:*`, `risk:*`, `blocker:*`, `task:*`, `package:*`, `repo:*`, and + `automation:*`. +7. Keep every node and edge compact and source-traced. +8. Update `catalog.json` counts and health. +9. Validate again. +10. Write `memory/maintenance/pam-daily-maintenance-YYYY-MM-DD.md` with graph + changes, highlights, skipped candidates, blockers, validation status, and next + action. + +## Highlights + +Every scheduled report should include a short highlights section summarizing the +important memories found during the scan. Highlights are for the human reader, +not for schema bookkeeping. + +Good highlights are: + +- 3-7 bullets; +- public-safe and non-sensitive; +- focused on durable meaning: decisions, blockers, risks, project state, + published artifacts, or changed operational posture; +- sourced by the same scanned memory used for graph promotion. + +Avoid highlights that expose raw private messages, secrets, webhook values, full +paths to secret stores, or noisy implementation minutiae. If nothing meaningful +was found, say that directly and name the bounded scan window. + +## Safety rules + +- Do not store secrets, credentials, tokens, webhook URLs, cookies, private + keys, raw private chat, or secret-store paths. +- Prefer source pointers over copied detail. +- Treat chronological logs and transcripts as source material, not graph content + by default. +- In shared contexts, avoid private profile/preference details unless the user + explicitly approves them. +- If validation fails, fix only PAM-owned files or roll back the PAM-owned + change and report `BLOCKED`. + +## Report format + +Use a concise human-facing report. Example: + +```text +PAM daily maintenance — YYYY-MM-DD HH:mm UTC +Status: 🟢 OK | 🟡 WARNING | 🔴 BLOCKED + +Highlights: +- Approval policy was promoted as an active project memory. +- Chat ingestion remains blocked by platform access. +- A runtime security posture item remains a P1 risk. + +Checks: +- 🟢 Graph validation — passed, 0 errors / 0 warnings. +- 🟢 Graph updates — added 3 nodes, 4 edges, 2 aliases. +- 🟡 Recent memory scan — skipped 2 private or unstable candidates. +- 🟢 Safety — wrote only PAM-owned paths; no secrets or raw chat stored. +- Next action: review `blocker:example` before tomorrow's run. +``` + +Do not make the daily report an acceptance-criteria-only checklist. Acceptance +criteria are secondary health evidence; real graph maintenance is the primary +job. diff --git a/memory/agent-memory/pam-openclaw.md b/memory/agent-memory/pam-openclaw.md index b645c7e..f6da1ea 100644 --- a/memory/agent-memory/pam-openclaw.md +++ b/memory/agent-memory/pam-openclaw.md @@ -135,6 +135,35 @@ If uncertain, report `unknown` or `partial`; do not force OpenClaw specializatio 9. Report the PAM installation acceptance criteria one by one before claiming completion. +## Runtime-native graph maintenance + +When OpenClaw cron or another runtime-native scheduler is configured for PAM, +the recurring job SHOULD perform real graph maintenance, not only repeat the +installation acceptance criteria. + +Daily or near-daily maintenance should: + +1. Validate `memory/graph/*.jsonl`. +2. Read the latest PAM maintenance report to identify the previous state. +3. Do a bounded scan of recent durable memory, usually the last 24-48 hours or + since the previous maintenance run. +4. Promote only stable, useful, non-sensitive records into PAM-owned graph files. +5. Prefer entities and relations such as projects, systems, decisions, tasks, + risks, blockers, packages, repositories, policies, dependencies, and + automations. +6. Keep every graph record compact and source-traced. +7. Update `catalog.json`, validate again, and write a concise report under + `memory/maintenance/`. +8. Include a short highlights section with 3-7 public-safe bullets summarizing + the most important memories found, promoted, skipped, or blocked. + +The daily report may include acceptance/status checks, but those checks are +secondary. It should also include human-readable highlights. If no graph +candidates were found, the report should say what was scanned and why nothing +was promoted. + +For a reusable job runbook, see `docs/openclaw-daily-graph-maintenance.md`. + ## OpenClaw-specific installation acceptance criteria In addition to the generic PAM installation criteria, an OpenClaw workspace must diff --git a/memory/agent-memory/pam-runtime.md b/memory/agent-memory/pam-runtime.md index cb2848d..d25ab0a 100644 --- a/memory/agent-memory/pam-runtime.md +++ b/memory/agent-memory/pam-runtime.md @@ -16,6 +16,29 @@ completion. 6. If `memory/agent-memory/pam-openclaw.md` exists and the workspace has OpenClaw-style memory, read it before proposing memory writes or migrations. +## File-Only Efficiency Gate + +When the agent has only file read/search tools, PAM should still reduce +retrieval cost. For memory, project, task, decision, or procedure questions: + +- Use the graph-first path above before scanning broad markdown corpora. +- Keep initial routing to `memory/pam.version.json`, `memory/graph/catalog.json`, + `aliases.jsonl`, `nodes.jsonl`, and `edges.jsonl`. +- Follow at most one source file from the best matching node before falling + back to a wider search. +- Treat missing aliases, stale statuses, and broad fallback as coverage gaps to + record in the next maintenance pass. + +Validate this behavior with: + +```bash +npm run memory:graph:coverage -- --json +``` + +Default acceptance target: at least 80% of realistic queries should route to +the expected node with no more than 5 graph files and 100 KB read before source +fallback. + ## Direct JSONL Fallback Node tools are optional. Without Node, use any text search or JSONL parser: diff --git a/memory/graph/aliases.jsonl b/memory/graph/aliases.jsonl index e3c7827..0fe8bca 100644 --- a/memory/graph/aliases.jsonl +++ b/memory/graph/aliases.jsonl @@ -7,6 +7,11 @@ {"a":"maintenance cli","id":"tool:maintenance"} {"a":"graph cli","id":"tool:graph"} {"a":"benchmark cli","id":"tool:benchmark"} +{"a":"file-only coverage","id":"tool:file-only-coverage"} +{"a":"coverage cli","id":"tool:file-only-coverage"} +{"a":"file-only agent benchmark","id":"tool:file-only-coverage"} {"a":"markdown migration","id":"migration:markdown-to-graph-v1"} +{"a":"0.5.0 migration","id":"migration:0.5-file-only-coverage"} +{"a":"PAM 0.5.0","id":"migration:0.5-file-only-coverage"} {"a":"agent-facing runbooks","id":"principle:agent-facing-runbooks"} {"a":"runbook quality gate","id":"principle:agent-facing-runbooks"} diff --git a/memory/graph/catalog.json b/memory/graph/catalog.json index bb3dd31..eba21d8 100644 --- a/memory/graph/catalog.json +++ b/memory/graph/catalog.json @@ -1,9 +1,9 @@ { "schemaVersion": "pam-graph-v1", - "generatedAt": "2026-05-21T18:13:47.933Z", - "nodeCount": 12, - "edgeCount": 13, - "aliasCount": 12, + "generatedAt": "2026-05-22T16:17:27.591Z", + "nodeCount": 14, + "edgeCount": 17, + "aliasCount": 17, "sourceFiles": [ "memory/graph/nodes.jsonl", "memory/graph/edges.jsonl", diff --git a/memory/graph/edges.jsonl b/memory/graph/edges.jsonl index b069b1b..144abe1 100644 --- a/memory/graph/edges.jsonl +++ b/memory/graph/edges.jsonl @@ -5,9 +5,13 @@ {"f":"project:pam","r":"has-tool","t":"tool:maintenance","st":"confirmed","c":"high","u":"2026-05-05","src":"package.json"} {"f":"project:pam","r":"has-tool","t":"tool:graph","st":"confirmed","c":"high","u":"2026-05-05","src":"package.json"} {"f":"project:pam","r":"has-tool","t":"tool:benchmark","st":"confirmed","c":"high","u":"2026-05-05","src":"package.json"} +{"f":"project:pam","r":"has-tool","t":"tool:file-only-coverage","st":"confirmed","c":"high","u":"2026-05-22","src":"package.json"} +{"f":"tool:file-only-coverage","r":"tests-format","t":"format:graph-v1","st":"confirmed","c":"high","u":"2026-05-22","src":"tools/memory-graph.mjs"} {"f":"project:pam","r":"keeps-compatible-log","t":"log:conversation","st":"confirmed","c":"high","u":"2026-05-05","src":"memory/index.md"} {"f":"project:pam","r":"keeps-compatible-log","t":"log:knowledge","st":"confirmed","c":"high","u":"2026-05-05","src":"memory/index.md"} {"f":"format:graph-v1","r":"migrates-from","t":"migration:markdown-to-graph-v1","st":"confirmed","c":"high","u":"2026-05-05","src":"migrations/markdown-v0-to-graph-v1.md"} +{"f":"format:graph-v1","r":"keeps-schema-for","t":"migration:0.5-file-only-coverage","st":"confirmed","c":"high","u":"2026-05-22","src":"migrations/0.4.0-to-0.5.0-file-only-coverage.md"} {"f":"doc:runtime","r":"prefers","t":"format:graph-v1","st":"confirmed","c":"high","u":"2026-05-05","src":"memory/agent-memory/pam-runtime.md"} +{"f":"doc:runtime","r":"recommends","t":"tool:file-only-coverage","st":"confirmed","c":"high","u":"2026-05-22","src":"memory/agent-memory/pam-runtime.md"} {"f":"project:pam","r":"has-principle","t":"principle:agent-facing-runbooks","st":"confirmed","c":"high","u":"2026-05-07","src":"memory/agent-memory/pam.md"} {"f":"doc:pam-contract","r":"defines-principle","t":"principle:agent-facing-runbooks","st":"confirmed","c":"high","u":"2026-05-07","src":"memory/agent-memory/pam.md"} diff --git a/memory/graph/nodes.jsonl b/memory/graph/nodes.jsonl index 71b5c81..a76586b 100644 --- a/memory/graph/nodes.jsonl +++ b/memory/graph/nodes.jsonl @@ -5,8 +5,10 @@ {"id":"tool:maintenance","k":"tool","n":"Memory maintenance CLI","d":"Optional Node tool for rotating logs, rebuilding archive indexes, and bounded synthesis.","st":"confirmed","c":"high","u":"2026-05-05","src":"tools/memory-maintenance.mjs","tags":["node","optional"]} {"id":"tool:graph","k":"tool","n":"Graph memory CLI","d":"Optional Node tool for validating, querying, indexing, and measuring graph JSONL memory.","st":"confirmed","c":"high","u":"2026-05-05","src":"tools/memory-graph.mjs","tags":["node","optional","graph"]} {"id":"tool:benchmark","k":"tool","n":"Memory benchmark CLI","d":"Optional Node tool for anonymous read-volume and command-duration memory benchmarks.","st":"confirmed","c":"high","u":"2026-05-05","src":"tools/memory-benchmark.mjs","tags":["node","optional","benchmark"]} +{"id":"tool:file-only-coverage","k":"tool","n":"File-only coverage CLI","d":"Checks graph-first retrieval cost and expected node hits for agents without semantic memory search.","st":"confirmed","c":"high","u":"2026-05-22","src":"tools/memory-graph.mjs","tags":["node","optional","coverage","file-only"]} {"id":"format:graph-v1","k":"format","n":"Graph JSONL v1","d":"AI-first JSONL nodes, edges, aliases, and catalog for compact direct retrieval.","st":"confirmed","c":"high","u":"2026-05-05","src":"memory/pam.version.json","tags":["graph","jsonl"]} {"id":"log:conversation","k":"log","n":"Conversation log","d":"Compact chronological work log retained for markdown compatibility.","st":"confirmed","c":"high","u":"2026-05-05","src":"memory/conversation-log.md","tags":["markdown","log"]} {"id":"log:knowledge","k":"log","n":"Knowledge log","d":"Durable facts and raw discoveries retained for markdown compatibility.","st":"confirmed","c":"high","u":"2026-05-05","src":"memory/knowledge-log.md","tags":["markdown","log"]} {"id":"migration:markdown-to-graph-v1","k":"migration","n":"Markdown to graph-v1 migration","d":"Guide for markdown-only PAM users adopting graph-v1 without losing portable markdown memory.","st":"confirmed","c":"high","u":"2026-05-05","src":"migrations/markdown-v0-to-graph-v1.md","tags":["migration"]} +{"id":"migration:0.5-file-only-coverage","k":"migration","n":"PAM 0.5.0 file-only coverage migration","d":"Upgrade guide for enabling file-only graph coverage without changing graph-v1 schema.","st":"confirmed","c":"high","u":"2026-05-22","src":"migrations/0.4.0-to-0.5.0-file-only-coverage.md","tags":["migration","coverage","file-only"]} {"id":"principle:agent-facing-runbooks","k":"principle","n":"Agent-facing runbooks","d":"Runbooks should use MUST/DO NOT rules, quality gates, examples, and checkable completion criteria.","st":"confirmed","c":"high","u":"2026-05-07","src":"memory/agent-memory/pam.md","tags":["runbook","agent","quality"]} diff --git a/memory/pam.version.json b/memory/pam.version.json index 730d685..a440b8e 100644 --- a/memory/pam.version.json +++ b/memory/pam.version.json @@ -1,5 +1,5 @@ { - "pamVersion": "0.4.0", + "pamVersion": "0.5.0", "memoryFormat": "graph-v1", "graphSchemaVersion": "pam-graph-v1", "features": { @@ -12,7 +12,8 @@ "mcpServer": true, "subagentCurator": true, "subagentScribe": true, - "proposalApply": true + "proposalApply": true, + "fileOnlyCoverage": true }, - "updated": "2026-05-28" + "updated": "2026-05-27" } diff --git a/migrations/0.4.0-to-0.5.0-file-only-coverage.md b/migrations/0.4.0-to-0.5.0-file-only-coverage.md new file mode 100644 index 0000000..1db2011 --- /dev/null +++ b/migrations/0.4.0-to-0.5.0-file-only-coverage.md @@ -0,0 +1,144 @@ +# Migration: PAM 0.4.0 to 0.5.0 file-only coverage + +Status: optional adoption guide +From: `0.4.0` +To: `0.5.0` +Graph schema change: none + +## Summary + +PAM 0.5.0 keeps the existing graph format: + +```json +{ + "memoryFormat": "graph-v1", + "graphSchemaVersion": "pam-graph-v1" +} +``` + +This migration adds a measurable file-only retrieval gate. It helps agents that +only have file read/search tools prove that PAM is reducing broad memory scans +instead of existing only as a parallel documentation structure. + +## When to use this guide + +Use this guide when: + +- an existing PAM 0.4.0 workspace wants to test graph-first retrieval cost; +- agents often lack semantic `memory_search` or vector recall; +- maintenance should promote concrete aliases and operational final states; +- you need a repeatable PASS/PARTIAL/BLOCKED report for PAM usefulness. + +Skip this guide when the workspace only wants static markdown memory and does +not need graph-first efficiency measurements. + +## Steps + +1. Update `memory/pam.version.json`: + +```json +{ + "pamVersion": "0.5.0", + "memoryFormat": "graph-v1", + "graphSchemaVersion": "pam-graph-v1", + "features": { + "fileOnlyCoverage": true + } +} +``` + +Preserve existing feature flags; add `fileOnlyCoverage` instead of replacing +the whole object. + +2. Install or update the runtime guide: + +```text +memory/agent-memory/pam-runtime.md +``` + +The guide should require graph-first lookup before broad corpus search for +memory, project, task, decision, and procedure questions. + +3. Add a file-only coverage scenario: + +```text +benchmarks/file-only-coverage.json +``` + +Start with 5 to 10 realistic queries. Each entry should include: + +```json +{ + "q": "runtime guide", + "expectedId": "doc:runtime" +} +``` + +4. Add or update local agent instructions such as: + +```text +AGENTS.md +CLAUDE.md +GEMINI.md +``` + +They should tell future agents: + +- PAM graph files are the first retrieval path; +- broad markdown scans are fallback behavior; +- coverage failures should become alias/node/source improvements; +- private raw text and secrets must not be copied into coverage reports. + +5. Validate the graph: + +```bash +npm run memory:graph:validate +``` + +6. Run file-only coverage: + +```bash +npm run memory:graph:coverage -- --json +``` + +7. Classify the result: + +- `PASS`: hit rate is at least 80%, initial graph routing reads no more than 5 + files / 100 KB, and each query opens no more than one targeted source file. +- `PARTIAL`: the graph is valid but some realistic queries require broad + fallback or resolve only through non-top results. +- `BLOCKED`: the graph is invalid, missing, or cannot be read with file tools. + +8. Feed coverage gaps into maintenance. + +For each failed or partial query, add safe source-traced aliases, node digests, +status markers, or edges. Prefer concrete operational terms users actually ask +for. Do not promote secrets, raw private conversations, credentials, or +unapproved personal data. + +## OpenClaw notes + +For OpenClaw workspaces, write only PAM-owned paths by default: + +```text +memory/pam.version.json +memory/agent-memory/pam-runtime.md +memory/graph/* +memory/maintenance/* +benchmarks/file-only-coverage.json +``` + +Do not rewrite `MEMORY.md`, existing OpenClaw memory corpus files, wiki pages, +or project-specific conventions unless the user explicitly approves. + +## Expected result + +A successful migration has: + +- unchanged graph schema (`pam-graph-v1`); +- `pamVersion` updated to `0.5.0`; +- `features.fileOnlyCoverage: true`; +- a file-only coverage scenario; +- local agent instructions that prefer graph-first retrieval; +- graph validation passing; +- coverage report classified as `PASS`, `PARTIAL`, or `BLOCKED`. diff --git a/package.json b/package.json index bd590bb..72b9ace 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "portable-agent-memory", - "version": "0.4.0", + "version": "0.5.0", "private": false, "description": "Markdown-first portable memory, wiki, and maintenance toolkit for AI agents.", "type": "module", @@ -9,6 +9,7 @@ "benchmark:current": "node tools/memory-benchmark.mjs current", "memory:detect": "node tools/memory-migration.mjs detect", "memory:graph:index": "node tools/memory-graph.mjs index", + "memory:graph:coverage": "node tools/memory-graph.mjs coverage", "memory:graph:query": "node tools/memory-graph.mjs query", "memory:graph:stats": "node tools/memory-graph.mjs stats", "memory:graph:validate": "node tools/memory-graph.mjs validate", diff --git a/tools/memory-graph.mjs b/tools/memory-graph.mjs index 4a2662d..70d2c3e 100644 --- a/tools/memory-graph.mjs +++ b/tools/memory-graph.mjs @@ -7,6 +7,13 @@ const __dirname = path.dirname(__filename); const WORKSPACE_ROOT = path.resolve(__dirname, ".."); const GRAPH_DIR = "memory/graph"; const MAX_NODE_DIGEST_CHARS = 180; +const DEFAULT_COVERAGE_SCENARIO = "benchmarks/file-only-coverage.json"; +const DEFAULT_FILE_ONLY_BUDGET = { + maxCoreFiles: 5, + maxCoreBytes: 100 * 1024, + maxSourceFilesPerQuery: 1, + minHitRate: 0.8 +}; function resolveWorkspacePath(workspaceRoot, relativePath) { return path.join(workspaceRoot, relativePath); @@ -205,6 +212,181 @@ function graphStats(workspaceRoot = WORKSPACE_ROOT) { }; } +function fileMetric(workspaceRoot, relativePath) { + const content = fs.readFileSync(resolveWorkspacePath(workspaceRoot, relativePath), "utf8"); + return { + path: relativePath, + bytes: Buffer.byteLength(content), + tokenProxy: Math.ceil(Buffer.byteLength(content) / 4) + }; +} + +function aggregateFileMetrics(workspaceRoot, files) { + const uniqueFiles = [...new Set(files)].filter(Boolean); + const existingFiles = uniqueFiles.filter((file) => fs.existsSync(resolveWorkspacePath(workspaceRoot, file))); + const missingFiles = uniqueFiles.filter((file) => !fs.existsSync(resolveWorkspacePath(workspaceRoot, file))); + const metrics = existingFiles.map((file) => fileMetric(workspaceRoot, file)); + return { + fileCount: metrics.length, + missingFileCount: missingFiles.length, + bytes: metrics.reduce((sum, entry) => sum + entry.bytes, 0), + tokenProxy: metrics.reduce((sum, entry) => sum + entry.tokenProxy, 0), + files: metrics, + missingFiles + }; +} + +function listFilesRecursive(workspaceRoot, relativeDir) { + const absoluteDir = resolveWorkspacePath(workspaceRoot, relativeDir); + if (!fs.existsSync(absoluteDir)) { + return []; + } + + const entries = fs.readdirSync(absoluteDir, { withFileTypes: true }); + return entries.flatMap((entry) => { + const relativePath = toPosixPath(path.join(relativeDir, entry.name)); + if (entry.isDirectory()) { + return listFilesRecursive(workspaceRoot, relativePath); + } + if (entry.isFile()) { + return [relativePath]; + } + return []; + }); +} + +function defaultCoverageQueries() { + return [ + { q: "PAM", expectedId: "project:pam" }, + { q: "runtime guide", expectedId: "doc:runtime" }, + { q: "graph cli", expectedId: "tool:graph" }, + { q: "markdown migration", expectedId: "migration:markdown-to-graph-v1" }, + { q: "agent-facing runbooks", expectedId: "principle:agent-facing-runbooks" } + ]; +} + +function readCoverageScenario(workspaceRoot, scenarioPath) { + const absolutePath = resolveWorkspacePath(workspaceRoot, scenarioPath); + if (!fs.existsSync(absolutePath)) { + return { + path: scenarioPath, + queries: defaultCoverageQueries() + }; + } + + const scenario = JSON.parse(fs.readFileSync(absolutePath, "utf8")); + if (!Array.isArray(scenario.queries)) { + throw new Error(`Coverage scenario must include a queries array: ${scenarioPath}`); + } + return { + path: scenarioPath, + name: scenario.name, + queries: scenario.queries + }; +} + +function collectFileOnlyCoverage(workspaceRoot = WORKSPACE_ROOT, options = {}) { + const graph = loadGraph(workspaceRoot); + const validation = validateGraph(graph); + const scenario = readCoverageScenario(workspaceRoot, options.scenario ?? DEFAULT_COVERAGE_SCENARIO); + const budget = { + maxCoreFiles: options.maxCoreFiles ?? DEFAULT_FILE_ONLY_BUDGET.maxCoreFiles, + maxCoreBytes: options.maxCoreBytes ?? DEFAULT_FILE_ONLY_BUDGET.maxCoreBytes, + maxSourceFilesPerQuery: options.maxSourceFilesPerQuery ?? DEFAULT_FILE_ONLY_BUDGET.maxSourceFilesPerQuery, + minHitRate: options.minHitRate ?? DEFAULT_FILE_ONLY_BUDGET.minHitRate + }; + const coreFiles = [ + "memory/pam.version.json", + `${GRAPH_DIR}/catalog.json`, + `${GRAPH_DIR}/aliases.jsonl`, + `${GRAPH_DIR}/nodes.jsonl`, + `${GRAPH_DIR}/edges.jsonl` + ]; + const corpusFiles = [ + "AGENT_BOOTSTRAP.md", + "README.md", + ...listFilesRecursive(workspaceRoot, "memory").filter((file) => /\.(json|jsonl|md)$/i.test(file)) + ]; + const coreRead = aggregateFileMetrics(workspaceRoot, coreFiles); + const corpusRead = aggregateFileMetrics(workspaceRoot, corpusFiles); + + const queryResults = scenario.queries.map((entry) => { + const query = entry.q ?? entry.query ?? ""; + const expectedId = entry.expectedId ?? null; + const result = queryGraph(graph, { query, limit: 3 }); + const resultIds = result.results.map((node) => node.id); + const topResult = result.results[0] ?? null; + const targetSources = result.results.map((node) => node.src); + const sourceRead = aggregateFileMetrics(workspaceRoot, targetSources.slice(0, budget.maxSourceFilesPerQuery)); + const expectedMatched = expectedId ? resultIds.includes(expectedId) : resultIds.length > 0; + const topMatched = expectedId ? topResult?.id === expectedId : Boolean(topResult); + const sourceReadable = result.results.length === 0 || sourceRead.missingFileCount === 0; + const status = !sourceReadable ? "BLOCKED" : topMatched ? "PASS" : expectedMatched ? "PARTIAL" : "BLOCKED"; + + return { + query, + expectedId, + aliasResolvedTo: result.aliasResolvedTo, + resultIds, + topId: topResult?.id ?? null, + status, + sourceRead, + notes: { + opensRawSourceText: false, + candidateSourceCount: new Set(result.results.map((node) => node.src)).size + } + }; + }); + + const passCount = queryResults.filter((entry) => entry.status === "PASS").length; + const partialCount = queryResults.filter((entry) => entry.status === "PARTIAL").length; + const blockedCount = queryResults.filter((entry) => entry.status === "BLOCKED").length; + const hitRate = queryResults.length === 0 ? 0 : passCount / queryResults.length; + const coreBudgetOk = coreRead.fileCount <= budget.maxCoreFiles && coreRead.bytes <= budget.maxCoreBytes; + const sourceBudgetOk = queryResults.every((entry) => entry.sourceRead.fileCount <= budget.maxSourceFilesPerQuery); + const missingSourceCount = queryResults.reduce((sum, entry) => sum + entry.sourceRead.missingFileCount, 0); + const sourceFilesOk = missingSourceCount === 0; + const ok = validation.ok && coreBudgetOk && sourceBudgetOk && sourceFilesOk && hitRate >= budget.minHitRate; + + return { + coverageVersion: 1, + generatedAt: options.generatedAt ?? new Date().toISOString(), + privacy: { + aggregateOnly: true, + rawTextIncluded: false, + absolutePathsIncluded: false + }, + scenario: { + path: scenario.path, + name: scenario.name ?? null, + queryCount: queryResults.length + }, + budget, + graph: { + nodeCount: graph.nodes.length, + edgeCount: graph.edges.length, + aliasCount: graph.aliases.length, + valid: validation.ok + }, + readVolume: { + pamFirstCore: coreRead, + corpusFirst: corpusRead + }, + results: queryResults, + summary: { + ok, + passCount, + partialCount, + blockedCount, + hitRate, + coreBudgetOk, + sourceBudgetOk, + sourceFilesOk, + missingSourceCount + } + }; +} + function parseArgs(argv) { const args = [...argv]; const command = args.shift() ?? "validate"; @@ -222,6 +404,18 @@ function parseArgs(argv) { options.relation = args.shift() ?? ""; } else if (arg === "--limit") { options.limit = Number(args.shift() ?? "10"); + } else if (arg === "--scenario") { + options.scenario = args.shift(); + } else if (arg === "--max-files") { + options.maxCoreFiles = Number(args.shift() ?? DEFAULT_FILE_ONLY_BUDGET.maxCoreFiles); + } else if (arg === "--max-bytes") { + options.maxCoreBytes = Number(args.shift() ?? DEFAULT_FILE_ONLY_BUDGET.maxCoreBytes); + } else if (arg === "--max-source-files") { + options.maxSourceFilesPerQuery = Number(args.shift() ?? DEFAULT_FILE_ONLY_BUDGET.maxSourceFilesPerQuery); + } else if (arg === "--min-hit-rate") { + options.minHitRate = Number(args.shift() ?? DEFAULT_FILE_ONLY_BUDGET.minHitRate); + } else if (arg === "--generated-at") { + options.generatedAt = args.shift(); } } @@ -238,6 +432,7 @@ function print(data, json = false) { export { buildCatalog, + collectFileOnlyCoverage, graphStats, loadGraph, queryGraph, @@ -266,6 +461,15 @@ function main() { return; } + if (options.command === "coverage") { + const result = collectFileOnlyCoverage(WORKSPACE_ROOT, options); + print(result, options.json); + if (!result.summary.ok) { + process.exitCode = 1; + } + return; + } + if (options.command === "index") { const catalog = buildCatalog(); const catalogPath = resolveWorkspacePath(WORKSPACE_ROOT, `${GRAPH_DIR}/catalog.json`); diff --git a/tools/test-memory-graph.mjs b/tools/test-memory-graph.mjs index 5c58088..9755dfd 100644 --- a/tools/test-memory-graph.mjs +++ b/tools/test-memory-graph.mjs @@ -6,6 +6,7 @@ import test from "node:test"; import { buildCatalog, + collectFileOnlyCoverage, loadGraph, queryGraph, validateGraph @@ -84,3 +85,174 @@ test("graph catalog reports counts and health", () => { assert.equal(catalog.aliasCount, 1); assert.equal(catalog.health.status, "valid"); }); + +test("file-only coverage measures graph-first read volume without raw source text", () => { + const root = makeGraphWorkspace(); + fs.mkdirSync(path.join(root, "benchmarks"), { recursive: true }); + fs.writeFileSync( + path.join(root, "memory", "pam.version.json"), + '{"pamVersion":"0.4.0","memoryFormat":"graph-v1","graphSchemaVersion":"pam-graph-v1"}\n', + "utf8" + ); + fs.writeFileSync( + path.join(root, "memory", "graph", "catalog.json"), + JSON.stringify(buildCatalog(root, { generatedAt: "2026-05-05T00:00:00.000Z" })) + "\n", + "utf8" + ); + fs.writeFileSync( + path.join(root, "README.md"), + "Portable Agent Memory test corpus.\n", + "utf8" + ); + fs.writeFileSync( + path.join(root, "AGENT_BOOTSTRAP.md"), + "Bootstrap instructions for tests.\n", + "utf8" + ); + fs.writeFileSync( + path.join(root, "benchmarks", "file-only-coverage.json"), + JSON.stringify({ + queries: [ + { + q: "PAM", + expectedId: "project:pam" + } + ] + }) + "\n", + "utf8" + ); + + const coverage = collectFileOnlyCoverage(root, { + scenario: "benchmarks/file-only-coverage.json", + generatedAt: "2026-05-05T00:00:00.000Z" + }); + const serialized = JSON.stringify(coverage); + + assert.equal(coverage.summary.ok, true); + assert.equal(coverage.summary.blockedCount, 0); + assert.equal(coverage.privacy.rawTextIncluded, false); + assert.equal(coverage.readVolume.pamFirstCore.fileCount, 5); + assert.ok(coverage.readVolume.pamFirstCore.bytes > 0); + assert.ok(!serialized.includes(root)); + assert.ok(!serialized.includes("Portable Agent Memory test corpus.")); +}); + +test("file-only coverage classifies missing expected nodes as blocked", () => { + const root = makeGraphWorkspace(); + fs.mkdirSync(path.join(root, "benchmarks"), { recursive: true }); + fs.writeFileSync( + path.join(root, "memory", "pam.version.json"), + '{"pamVersion":"0.4.0","memoryFormat":"graph-v1","graphSchemaVersion":"pam-graph-v1"}\n', + "utf8" + ); + fs.writeFileSync( + path.join(root, "memory", "graph", "catalog.json"), + JSON.stringify(buildCatalog(root, { generatedAt: "2026-05-05T00:00:00.000Z" })) + "\n", + "utf8" + ); + fs.writeFileSync(path.join(root, "README.md"), "Portable Agent Memory test corpus.\n", "utf8"); + fs.writeFileSync(path.join(root, "AGENT_BOOTSTRAP.md"), "Bootstrap instructions for tests.\n", "utf8"); + fs.writeFileSync( + path.join(root, "benchmarks", "file-only-coverage.json"), + JSON.stringify({ + queries: [ + { + q: "unknown operational state", + expectedId: "state:missing" + } + ] + }) + "\n", + "utf8" + ); + + const coverage = collectFileOnlyCoverage(root, { + scenario: "benchmarks/file-only-coverage.json", + generatedAt: "2026-05-05T00:00:00.000Z" + }); + + assert.equal(coverage.summary.ok, false); + assert.equal(coverage.summary.blockedCount, 1); + assert.equal(coverage.results[0].status, "BLOCKED"); +}); + +test("file-only coverage blocks matched nodes with missing source files", () => { + const root = makeGraphWorkspace(); + fs.mkdirSync(path.join(root, "benchmarks"), { recursive: true }); + fs.writeFileSync( + path.join(root, "memory", "pam.version.json"), + '{"pamVersion":"0.4.0","memoryFormat":"graph-v1","graphSchemaVersion":"pam-graph-v1"}\n', + "utf8" + ); + fs.writeFileSync( + path.join(root, "memory", "graph", "catalog.json"), + JSON.stringify(buildCatalog(root, { generatedAt: "2026-05-05T00:00:00.000Z" })) + "\n", + "utf8" + ); + fs.writeFileSync(path.join(root, "README.md"), "Portable Agent Memory test corpus.\n", "utf8"); + fs.writeFileSync(path.join(root, "AGENT_BOOTSTRAP.md"), "Bootstrap instructions for tests.\n", "utf8"); + fs.writeFileSync( + path.join(root, "benchmarks", "file-only-coverage.json"), + JSON.stringify({ + queries: [ + { + q: "graph cli", + expectedId: "tool:graph" + } + ] + }) + "\n", + "utf8" + ); + + const coverage = collectFileOnlyCoverage(root, { + scenario: "benchmarks/file-only-coverage.json", + generatedAt: "2026-05-05T00:00:00.000Z" + }); + + assert.equal(coverage.summary.ok, false); + assert.equal(coverage.summary.blockedCount, 1); + assert.equal(coverage.summary.missingSourceCount, 1); + assert.equal(coverage.summary.sourceFilesOk, false); + assert.equal(coverage.results[0].status, "BLOCKED"); + assert.deepEqual(coverage.results[0].sourceRead.missingFiles, ["tools/memory-graph.mjs"]); +}); + +test("file-only coverage reads returned candidate sources instead of expected target sources", () => { + const root = makeGraphWorkspace(); + fs.mkdirSync(path.join(root, "benchmarks"), { recursive: true }); + fs.mkdirSync(path.join(root, "tools"), { recursive: true }); + fs.writeFileSync( + path.join(root, "memory", "pam.version.json"), + '{"pamVersion":"0.4.0","memoryFormat":"graph-v1","graphSchemaVersion":"pam-graph-v1"}\n', + "utf8" + ); + fs.writeFileSync( + path.join(root, "memory", "graph", "catalog.json"), + JSON.stringify(buildCatalog(root, { generatedAt: "2026-05-05T00:00:00.000Z" })) + "\n", + "utf8" + ); + fs.writeFileSync(path.join(root, "README.md"), "Portable Agent Memory test corpus.\n", "utf8"); + fs.writeFileSync(path.join(root, "AGENT_BOOTSTRAP.md"), "Bootstrap instructions for tests.\n", "utf8"); + fs.writeFileSync(path.join(root, "tools", "memory-graph.mjs"), "console.log('graph cli');\n", "utf8"); + fs.writeFileSync( + path.join(root, "benchmarks", "file-only-coverage.json"), + JSON.stringify({ + queries: [ + { + q: "graph cli", + expectedId: "project:pam" + } + ] + }) + "\n", + "utf8" + ); + + const coverage = collectFileOnlyCoverage(root, { + scenario: "benchmarks/file-only-coverage.json", + generatedAt: "2026-05-05T00:00:00.000Z" + }); + + assert.equal(coverage.summary.ok, false); + assert.equal(coverage.results[0].status, "BLOCKED"); + assert.equal(coverage.results[0].topId, "tool:graph"); + assert.equal(coverage.results[0].sourceRead.files[0].path, "tools/memory-graph.mjs"); +});