diff --git a/.agents/AGENTS.md b/.agents/AGENTS.md index 70b1ed6bd..3f68c68f1 100644 --- a/.agents/AGENTS.md +++ b/.agents/AGENTS.md @@ -8,43 +8,62 @@ paths: # CodeGraph System Instructions -This project is backed by a custom **CodeGraph MCP server**. CodeGraph maintains a local Tree-sitter knowledge graph encompassing every symbol, edge, boundary, and file within this workspace. Reads operate at sub-millisecond speeds and deliver accurate structural insights that traditional text-based tools (like grep) cannot match. +This project is backed by a **CodeGraph MCP server** — a local tree-sitter +semantic graph of every symbol and call chain in the workspace. Reads are +sub-millisecond and return structural information grep cannot match. --- ## 🚨 CRITICAL CONSTRAINTS (Read First) -- **NEVER use generic text-search, grep, or file-reading tools** if a symbol, reference, or definition can be located using CodeGraph. -- **DO NOT double-check or re-verify** CodeGraph results with native file reads. Treat the knowledge graph as the absolute, single source of truth for codebase architecture. -- **Handle uninitialized states immediately:** If any CodeGraph tool returns a `"not initialized"` or missing index error, **STOP execution immediately** and instruct the user to run `codegraph init -i` in their terminal. Do not attempt to parse or scan the codebase manually to compensate. -- **Minimize token overhead:** Prefer targeted structural queries over dumping entire file contents into the context window. +- **NEVER use generic text-search, grep, or file-reading tools** when a + symbol, reference, or definition can be located with CodeGraph. +- **Do NOT re-verify** CodeGraph results with file reads — the graph is the + single source of truth for codebase structure. +- **Handle unbound sessions:** query tools refuse until the session is bound. + Call `codegraph_init {"path": ...}` (non-blocking, does NOT index), then + `codegraph_index {}` to build/refresh the index. +- **Minimize token overhead:** prefer targeted structural queries and + `id`-based lookups over dumping file contents into context. --- ## 🛠️ Tool Selection Guide -Always prefer `codegraph` tools for **structural** questions — tracing call hierarchies, mapping dependencies, determining definitions, and verifying signatures. Use standard filesystem tools *only* for literal text queries or applying actual code edits. +Prefer codegraph for **structural** questions. Use filesystem tools only for +literal text queries or applying edits. | Intent / Question | Recommended MCP Tool | | :--- | :--- | -| *"Where is symbol X defined?"* | `codegraph_search` | -| *"What callers invoke function Y?"* | `codegraph_callers` | -| *"What methods or functions does Y call?"* | `codegraph_callees` | -| *"What components or files will break if I modify Z?"* | `codegraph_impact` | -| *"Show me Y's exact signature and internal block"* | `codegraph_node` | -| *"Give me focused, aggregated context for this task"* | `codegraph_context` | -| *"What files exist under a specific path/ directory?"* | `codegraph_files` | -| *"Is the local knowledge graph healthy and active?"* | `codegraph_status` | +| *"Where is symbol X defined?"* | `codegraph_search_symbol` (contains/prefix/suffix/exact) | +| *"Show me this symbol by id / exact name"* | `codegraph_symbol` | +| *"What calls function Y?"* | `codegraph_callers` | +| *"What does Y call directly?"* | `codegraph_callees` | +| *"What breaks if I modify Z?"* | `codegraph_impact` | +| *"Show me Y's call chain"* | `codegraph_flow` | +| *"Find flows containing a pattern (loop + call)"* | `codegraph_search_flow` | +| *"Give me focused, aggregated context for a task"* | `codegraph_context` | +| *"Who calls the library function foo?"* | `codegraph_references` | +| *"What fields/methods does class C have?"* | `codegraph_class` | +| *"Which symbols are annotated @X?"* | `codegraph_search_by_annotation` | +| *"What files exist under path/?"* | `codegraph_files` | +| *"Is the index healthy?"* | `codegraph_status` | +| *"What does this MR change in the graph?"* | `codegraph_diff` | +| *"Simulate a flow's behavior with mocks"* | `codegraph_sandbox` / `codegraph_diff_simulate` | --- -## 💡 Rules of Thumb & Workflows +## 💡 Rules of Thumb -### 1. Unified Context Gathering -Do not chain manual searches and individual node inspections yourself. **`codegraph_context` is designed to perform aggregate lookups in a single call.** Execute it first when onboarding onto a new task or analyzing a localized bug. +1. **`codegraph_context` first** — it aggregates search + callers + callees + in one call; don't chain searches manually. +2. **Query impact before editing** — `codegraph_impact` pinpoints downstream + effects so you only touch relevant files. +3. **Trust the results** — AST-derived. If a lookup yields nothing, the + symbol is not in the active workspace index. +4. **Duplicate names** → the tool returns `ambiguous: true` with matches; + retry with the numeric `id` alone. -### 2. Defensive Token Preservation -Before updating an API route, a UI component, or a system utility, query `codegraph_impact` to pinpoint downstream effects. This ensures you only request and modify files strictly relevant to the current objective, preventing context window saturation. - -### 3. Strict AST Reliance -Because CodeGraph parses the Abstract Syntax Tree (AST), its structural insights are guaranteed. If a symbol look-up yields no results, assume the symbol does not exist in the current active workspace index. \ No newline at end of file +The full, always-current guide (timeout/resume protocol, response formats, +sandbox contracts) ships inside the binary as the server instructions — see +`docs/codegraph.md` in the CodeGraph repository. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 88c57e1ec..22bd121bd 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -11,6 +11,18 @@ env: RUSTFLAGS: -D warnings jobs: + sonarqube: + name: SonarQube + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + with: + fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis + - name: SonarQube Scan + uses: SonarSource/sonarqube-scan-action@7006c4492b2e0ee0f816d36501671557c97f5995 # v8.1.0 + env: + SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }} + clippy: name: clippy runs-on: ubuntu-latest diff --git a/Cargo.lock b/Cargo.lock index a4364f6ef..6b64c0341 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -766,6 +766,27 @@ dependencies = [ "toml", ] +[[package]] +name = "codesmell" +version = "1.2.0" +dependencies = [ + "anyhow", + "camino", + "clap", + "codegraph-core", + "codegraph-extract", + "codegraph-graph", + "dirs 5.0.1", + "globset", + "regex", + "rhai", + "serde", + "serde_json", + "tempfile", + "tokio", + "toml", +] + [[package]] name = "codspeed" version = "5.0.1" diff --git a/Cargo.toml b/Cargo.toml index 25597bfe6..099cba993 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -11,6 +11,7 @@ members = [ "crates/codegraph-bench", "crates/codegraph-installer", "crates/codegraph", + "crates/codesmell", ] [workspace.package] diff --git a/README.md b/README.md index 2d9b36c58..b814d0dcc 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,26 @@ runs the MCP server. All reading/interacting goes through MCP tools. Global flag `--path ` overrides the workspace root. +## CodeSmell — convention linter for LLM agents + +The workspace also ships [`codesmell`](docs/codesmell.md), a team-convention +linter built on CodeGraph facts. LLM agents read the conventions pack before +writing code and fix every reported violation afterwards — each violation +carries a `fix_hint`, so the agent repairs code instead of guessing: + +```bash +codesmell init # write .codesmell/policy.toml +codesmell guide # conventions pack for the LLM / team +codesmell check # lint the repo (style, architecture, testing) +git diff | codesmell check --diff - # change-aware validation +``` + +Policies cover function size/parameter/nesting limits, naming conventions +(`*Service`, `Async` suffix), layer boundaries (`controller !-> repository`, +severity `blocking`), and required unit tests for changed business logic +(severity `required`), with per-area `[[override]]` scopes. See +[docs/codesmell.md](docs/codesmell.md) for the full policy schema. + ## Supported languages 14 languages with full tree-sitter extraction + marker/chain walkers: @@ -172,7 +192,7 @@ report, plus the session tools `codegraph_init` / `codegraph_deinit` / | `codegraph_sandbox` | Compile a function group to machine code and run it against Rhai mocks | | `codegraph_diff` | Draft report of what an MR/patch would change in the graph | -Read the [server instructions](crates/codegraph-mcp/src/server-instructions.md) that ship with the binary — they tell your agent when to reach for which tool. +Read the [server instructions](docs/codegraph.md) that ship with the binary — they tell your agent when to reach for which tool. ### `codegraph_search_flow` pattern examples @@ -209,6 +229,7 @@ crates/ codegraph-mcp/ MCP server on the rmcp SDK (stdio + Streamable HTTP) + 27-tool dispatch, session-driven codegraph-bench/ Benchmarks (criterion search benches, storage benches, codspeed) codegraph/ CLI lifecycle (init/deinit/embed/serve --mcp) + watcher (notify + debounced full re-index) + codesmell/ Team-convention linter over in-memory CodeGraph facts (codesmell check / guide) ``` Pipeline: @@ -237,175 +258,31 @@ A `.codegraph/` directory is created next to your project: ``` .codegraph/ db.sqlite SQLite (WAL mode, single file — entities + radix streams); db.lmdb/ directory when the LMDB backend is selected - config.toml Language toggles, walker filters, storage backend, embedding settings + config.toml Languages, effect rules, storage backend, sandbox, embedding settings .gitignore Pre-filled so the index is never committed version Codegraph version that created the directory ``` -### config.toml example - -```toml -# Language toggles (all 14 enabled by default) -[languages] -rust = true -go = true -python = true -typescript = true -javascript = true -java = true -c = true -cpp = true -csharp = true -ruby = true -php = true -scala = true -swift = true -lua = true - -# Walker filters (same syntax as .gitignore) -[walker] -include = ["**/*"] -exclude = [ - ".git/**", - ".codegraph/**", - "target/**", - "node_modules/**", - "*.min.js", - "*.lock" -] - -# Storage backend — "sqlite" (default) | "lmdb" | "redis" | "memory" | "postgres" | "mysql" -[storage] -type = "sqlite" -# DSN override. Defaults: sqlite → sqlite:///.codegraph/db.sqlite, -# lmdb → lmdb:///.codegraph/db.lmdb (directory). Redis REQUIRES a dsn. -# dsn = "redis://localhost:6379" -# Postgres/MySQL use `dsns` (shard list) + `repo_id` — see below. - -# Semantic search (vector KNN) — OFF by default. See "Semantic search" below. -[embedding] -# backend = "fastembed" -# model = "bge-small-en-v1.5" -# cache_dir = "~/.cache/codegraph/embeddings" -``` - -### Storage backends - -The `[storage]` section selects where the index lives: +`config.toml` controls everything runtime-adjustable — highlights: -| `type` | Notes | +| Section | What it selects | |---|---| -| `sqlite` | Default. Single-file `db.sqlite` (WAL) inside `.codegraph/`. | -| `lmdb` | Memory-mapped KV (`db.lmdb/` directory inside `.codegraph/`). Same local-first workflow, mmap-friendly for large indexes. Enabled by default in the `codegraph` binary. | -| `redis` | Requires an explicit `dsn` (e.g. `redis://localhost:6379`) — there is no sensible local default. | -| `memory` | Ephemeral in-process index; nothing is persisted. | -| `postgres` / `mysql` | Multi-tenant, sharded — see below. | - -`dsn` (when set) overrides the derived default for any backend. - -### Postgres / MySQL (multi-tenant, sharded) +| `[languages]` | `headers = "auto" \| "c" \| "cpp"` — how `.h` files route between the C/C++ grammars | +| `[[effect_rules]]` | Call-name → `EffectType` classification, evaluated before the built-in defaults (first match wins) | +| `[storage]` | Backend: `sqlite` (default), `lmdb`, `redis`, `memory`, `postgres`/`mysql` (multi-tenant, sharded by `repo_id`) | +| `[sandbox]` | Behavior-sandbox defaults: `mock_dirs`, `loop_cap`, `branch_policy` | +| `[embedding]` | Opt-in semantic search (`fastembed`/`hashing`, model, cache dir, sqlite-vss, execution provider) | -CodeGraph can store the index in PostgreSQL or MySQL instead of the local -SQLite file. Every table is partitioned by a leading `repo_id` (a `u64` -partition key), so each project root (`.codegraph/`) maps to its own -partition — re-indexing or deleting one repo never touches another. Sharding -is `repo_id % N` across the configured DSN list. +Ignore handling: `.gitignore` + `.codegraphignore` are honored by the walker; +files ≥ 4 MiB or non-UTF-8 are skipped. -Build with the `rdbms` feature (it is **on by default** for the `codegraph` -binary): - -```bash -cargo build --features rdbms # default for `codegraph` -cargo build -p codegraph-mcp --features rdbms -``` - -`.codegraph/config.toml`: - -```toml -[storage] -type = "postgres" -# type = "mysql" -# Shard DSNs — shard = repo_id % len(dsns). One entry = single shard. -dsns = [ - "postgres://user:pass@db1:5432/codegraph", - "postgres://user:pass@db2:5432/codegraph", -] -# repo_id is generated automatically by `codegraph init` (self-heal) and -# written here. Do not edit it by hand. -# repo_id = 14028493579208694412 -``` - -**Schema is applied manually** — the binary does not run migrations. Run the -SQL files from `sql//` in order (currently `001-initial-schema.sql` -and `002-add-repos-registry.sql`) against every shard server before indexing: - -```bash -psql "$DSN" -f sql/postgres/001-initial-schema.sql -psql "$DSN" -f sql/postgres/002-add-repos-registry.sql -# mysql: -# mysql "$DB" < sql/mysql/001-initial-schema.sql -# mysql "$DB" < sql/mysql/002-add-repos-registry.sql -``` - -Then `codegraph init` (CLI) or `codegraph_init` (MCP tool) generates the -`repo_id` and stores the index on the right shard automatically. See -`sql/README.md` for the full multi-tenant + sharding design. - -### Semantic search (optional, opt-in) - -Vector similarity search over symbol embeddings is **off by default** — no -embedding model runs unless you enable it in config. The release binary -already bundles the fastembed (ONNX sentence-transformer) backend, so -enabling it is config-only — no rebuild required: - -1. Enable it in `.codegraph/config.toml`: - - ```toml - [embedding] - backend = "fastembed" # "hashing"/unset = off - model = "bge-small-en-v1.5" # 384-dim, default - cache_dir = "~/.cache/codegraph/embeddings" # global model cache (default) - # SQLite-only: point at a sqlite-vss (vector0/vss0) extension directory to - # run KNN through HNSW ANN inside the database: - # vss_extension = "~/.cache/codegraph/embeddings/vss" - # execution_provider = "coreml" # macOS hardware acceleration - ``` - -2. Optionally pre-download the model so indexing works offline: - - ```sh - codegraph embed --model bge-small-en-v1.5 - ``` - - The `codegraph embed` subcommand is compiled in when the binary is built - with `--features fastembed`. - -With embeddings enabled, `codegraph_search_symbol` gains the `match` modes -`"semantic"` (vector KNN — find symbols by similar/approximate names) and -`"hybrid"` (substring + semantic merged via Reciprocal Rank Fusion). Vectors -are persisted with the index, so restarts reuse them without re-embedding. - -Notes: -- If the model fails to load (no network, missing ONNX runtime), opening the - index **errors out** — there is no silent fallback to a lexical baseline. -- On macOS you can build with `--features fastembed,apple-accel` to run - embeddings on the Apple Neural Engine / GPU via the CoreML execution - provider. That feature is macOS-only and fails to build elsewhere. - -### C vs C++ headers (`.h`) - -By default, `.h` files are resolved automatically: -- **C++ project** (`.cpp`/`.hpp` present, no `.c`) → parsed as C++ -- **C project** (`.c` present, no C++ sources) → parsed as C -- **Mixed C/C++** → each `.h` inspected for C++ syntax (`namespace`, `class`, `template`, …) - -Override in `.codegraph/config.toml`: -```toml -[languages] -headers = "auto" # "auto" (default), "c", or "cpp" -``` +The full reference — every field, DSN derivation, sharding, the built-in +effect-rule table, embedding setup, and the file watcher — lives in +[docs/configuration.md](docs/configuration.md). Sandbox & mocking: +[docs/sandbox.md](docs/sandbox.md). MCP server & clients: +[docs/mcp.md](docs/mcp.md). All documentation is consolidated under +[docs/](docs/PLAN.md). -After changing this setting, run `codegraph init` (or call `codegraph_index` over MCP) to re-index headers. ## Why Rust? @@ -457,6 +334,7 @@ cargo test -p codegraph-mcp cargo test -p codegraph-sboxes # sandbox JIT: control flow + end-to-end traces cargo test -p codegraph-bench # pipeline integration cargo test -p codegraph-installer +cargo test -p codesmell # convention linter: policy, rules, fixtures ``` Feature flags on `codegraph-extract`: diff --git a/crates/codegraph-api/src/lib.rs b/crates/codegraph-api/src/lib.rs index 3d2f7e9ea..708bb0a9e 100644 --- a/crates/codegraph-api/src/lib.rs +++ b/crates/codegraph-api/src/lib.rs @@ -673,6 +673,15 @@ impl GraphApi { pub async fn stats(&self) -> codegraph_core::SemgraphStats { self.index().await.stats() } + + /// Stats đọc O(1) từ đĩa (không rebuild in-memory) — fallback `stats()` + /// nếu backend không hỗ trợ hoặc index cũ thiếu `sg_stats`. + pub async fn stats_cached(&self) -> codegraph_core::SemgraphStats { + match self.shared_index.stats_cached().await { + Some(s) => s, + None => self.stats().await, + } + } } /// Deadline từ `timeout_ms`: `0` = không giới hạn (None), `u64::MAX` diff --git a/crates/codegraph-extract/src/languages/common.rs b/crates/codegraph-extract/src/languages/common.rs index c785ae4b2..44da973b9 100644 --- a/crates/codegraph-extract/src/languages/common.rs +++ b/crates/codegraph-extract/src/languages/common.rs @@ -365,14 +365,41 @@ fn base_type_name(tn: &str) -> String { s.rsplit(['.', ':']).next().unwrap_or(s).trim().to_string() } +/// A node that directly or indirectly holds annotation leaves: either a known +/// wrapper container (`modifiers`, `decorators`, `attributes`, `attribute_list`, +/// `attribute_item`) or a leaf annotation kind itself. +fn is_annotation_container(kind: &str, kinds: &'static [&'static str]) -> bool { + matches!( + kind, + "modifiers" | "decorators" | "attributes" | "attribute_list" | "attribute_item" + ) || kinds.contains(&kind) +} + fn extract_annotations(node: &Node, src: &[u8], kinds: &'static [&'static str]) -> Vec { if kinds.is_empty() { return Vec::new(); } let mut out = Vec::new(); + // Attributes attached as children (Java modifiers, C#/PHP attribute_list, ...). for ch in named_children(node) { collect_annotation(&ch, src, kinds, &mut out); } + // Languages like Rust attach attributes as preceding sibling `attribute_item` + // nodes rather than as children of the declaration. Collect the contiguous + // run of attribute containers immediately before this symbol and stop at the + // first non-container sibling, so we don't grab another item's attributes. + if let Some(parent) = node.parent() { + let sibs = named_children(&parent); + if let Some(pos) = sibs.iter().position(|s| s.id() == node.id()) { + for sib in sibs[..pos].iter().rev() { + if is_annotation_container(sib.kind(), kinds) { + collect_annotation(sib, src, kinds, &mut out); + } else { + break; + } + } + } + } out } @@ -392,8 +419,8 @@ fn collect_annotation( let line = node.start_position().row as u32 + 1; out.push(Annotation { name, args, line }); } - // Wrapper node: Java modifiers, TS decorators, C# attributes... - if matches!(node.kind(), "modifiers" | "decorators" | "attributes") { + // Wrapper node: Java modifiers, TS decorators, C#/PHP attributes, Rust attribute_item... + if is_annotation_container(node.kind(), kinds) { for ch in named_children(node) { collect_annotation(&ch, src, kinds, out); } @@ -403,7 +430,11 @@ fn collect_annotation( fn annotation_args(node: &Node, src: &[u8]) -> HashMap { let mut args = HashMap::new(); for ch in named_children(node) { - if ch.kind() != "annotation_argument_list" { + // Java: `annotation_argument_list`; C#/PHP: `attribute_argument_list`. + if !matches!( + ch.kind(), + "annotation_argument_list" | "attribute_argument_list" + ) { continue; } for (i, arg) in named_children(&ch).into_iter().enumerate() { diff --git a/crates/codegraph-extract/src/languages/csharp.rs b/crates/codegraph-extract/src/languages/csharp.rs index 0ff458e2a..217c730d0 100644 --- a/crates/codegraph-extract/src/languages/csharp.rs +++ b/crates/codegraph-extract/src/languages/csharp.rs @@ -88,3 +88,104 @@ pub static SPEC: LangSpec = LangSpec { }; crate::lang_parser!(CSharpParser, SPEC); + +#[cfg(test)] +mod tests { + use crate::LangParser; + use codegraph_core::{Symbol, SymbolKind}; + + fn parse(src: &str) -> Vec { + super::CSharpParser::new() + .parse_file("test.cs", src) + .unwrap() + .symbols + } + + fn ann_names(sym: &Symbol) -> Vec { + sym.annotations.iter().map(|a| a.name.clone()).collect() + } + + #[test] + fn csharp_controller_annotations_are_extracted() { + let src = r#" +using Microsoft.AspNetCore.Mvc; + +namespace CodeGraphReproFixtures.Controllers; + +[ApiController] +[Route("api/[controller]")] +public class ProductsController : ControllerBase +{ + [HttpGet] + public IActionResult GetAll() => Ok(); + + [HttpGet("{id}")] + public IActionResult GetById(string id) => Ok(); + + [HttpPost()] + public IActionResult Create([FromBody] object body) => Ok(); + + [HttpPost("custom")] + public IActionResult CreateCustom([FromBody] object body) => Ok(); + + [HttpPut("{id}")] + public IActionResult Replace(string id, [FromBody] object body) => Ok(); + + [HttpPatch("{id}")] + public IActionResult PartialUpdate(string id, [FromBody] object body) => Ok(); + + [HttpDelete("{id}")] + public IActionResult Delete(string id) => Ok(); +} +"#; + let syms = parse(src); + let by_name = |n: &str| { + syms.iter() + .find(|s| s.name == n) + .unwrap_or_else(|| panic!("symbol `{n}` not found")) + .clone() + }; + + let cls = by_name("ProductsController"); + assert_eq!(cls.kind, SymbolKind::Class); + let cls_ann = ann_names(&cls); + assert!( + cls_ann.contains(&"ApiController".to_string()), + "class missing ApiController: {cls_ann:?}" + ); + assert!( + cls_ann.contains(&"Route".to_string()), + "class missing Route: {cls_ann:?}" + ); + + for (method, attr) in [ + ("GetAll", "HttpGet"), + ("GetById", "HttpGet"), + ("Create", "HttpPost"), + ("CreateCustom", "HttpPost"), + ("Replace", "HttpPut"), + ("PartialUpdate", "HttpPatch"), + ("Delete", "HttpDelete"), + ] { + let m = by_name(method); + assert_eq!(m.kind, SymbolKind::Method, "kind of {method}"); + let ann = ann_names(&m); + assert!( + ann.contains(&attr.to_string()), + "{method} missing {attr}: {ann:?}" + ); + } + + // Route argument template should be captured positionally. + let route = cls + .annotations + .iter() + .find(|a| a.name == "Route") + .expect("Route annotation"); + assert!( + route.args.values().any(|v| v.contains("api/[controller]")), + "route args: {:?}", + route.args + ); + } +} diff --git a/crates/codegraph-extract/src/languages/rust.rs b/crates/codegraph-extract/src/languages/rust.rs index 64afb29cc..6b3f3c55b 100644 --- a/crates/codegraph-extract/src/languages/rust.rs +++ b/crates/codegraph-extract/src/languages/rust.rs @@ -29,7 +29,7 @@ pub static SPEC: LangSpec = LangSpec { "mod_item", ], param_kinds: &[], - annotation_kinds: &[], + annotation_kinds: &["attribute"], // `impl Foo` không có name field — tên nằm ở field `type`. name_type_fallback: true, calls: &[CallRule { @@ -68,3 +68,57 @@ pub static SPEC: LangSpec = LangSpec { }; crate::lang_parser!(RustParser, SPEC); + +#[cfg(test)] +mod tests { + use crate::LangParser; + use codegraph_core::{Symbol, SymbolKind}; + + fn parse(src: &str) -> Vec { + super::RustParser::new() + .parse_file("test.rs", src) + .unwrap() + .symbols + } + + fn ann_names(sym: &Symbol) -> Vec { + sym.annotations.iter().map(|a| a.name.clone()).collect() + } + + #[test] + fn rust_attributes_are_extracted() { + let src = r#" +#[derive(Debug, Clone)] +pub struct Foo; + +#[tokio::main] +async fn main() {} +"#; + let syms = parse(src); + + let foo = syms + .iter() + .find(|s| s.name == "Foo") + .expect("Foo not found") + .clone(); + assert_eq!(foo.kind, SymbolKind::Class); + let ann = ann_names(&foo); + assert!( + ann.contains(&"derive".to_string()), + "Foo missing derive: {ann:?}" + ); + + let main = syms + .iter() + .find(|s| s.name == "main") + .expect("main not found") + .clone(); + assert_eq!(main.kind, SymbolKind::Function); + let ann = ann_names(&main); + // Rust attribute has no `name` field, so the first path identifier is used. + assert!( + ann.contains(&"tokio".to_string()), + "main missing tokio attribute: {ann:?}" + ); + } +} diff --git a/crates/codegraph-graph/src/lib.rs b/crates/codegraph-graph/src/lib.rs index a008a7add..df13b5a20 100644 --- a/crates/codegraph-graph/src/lib.rs +++ b/crates/codegraph-graph/src/lib.rs @@ -37,6 +37,7 @@ use crate::embeddings::{EmbeddingBackend, default_backend, embedding_enabled, make_backend}; pub use crate::search::Search; use crate::search::SearchResume; +use crate::storage::cached::CachedStorage; #[cfg(feature = "lmdb")] pub use crate::storage::lmdb::LmdbStorage; #[cfg(feature = "mysql")] @@ -45,7 +46,7 @@ pub use crate::storage::mysql::MySqlStorage; pub use crate::storage::postgres::PostgresStorage; #[cfg(feature = "sqlite")] pub use crate::storage::sqlite::SqliteStorage; -pub use crate::storage::{InMemoryStorage, Storage, Tx}; +pub use crate::storage::{InMemoryStorage, IndexCounts, Storage, Tx}; use crate::vector_index::VectorIndex; use codegraph_core::{ CallRecord, CallSite, CallSiteResult, ClassInfo, DependenciesReport, Dependency, EdgeMeta, @@ -63,6 +64,7 @@ use tokio::sync::RwLock; mod bloom; pub mod diff; pub mod embeddings; +mod lru; mod radix; mod search; mod shared; @@ -291,8 +293,7 @@ impl GraphIndex { /// (config chưa set → `[embedding].backend = "hashing"`), nên không load /// model. Nếu process đã set config fastembed trước đó mà model lỗi → panic. pub fn in_memory() -> Self { - let storage = Arc::new(RwLock::new(InMemoryStorage::default())) - as Arc>; + let storage: Box = Box::new(InMemoryStorage::default()); Self::new_with_storage(storage).expect("in_memory embedding backend init failed") } @@ -408,7 +409,7 @@ impl GraphIndex { let storage = crate::storage::lmdb::LmdbStorage::open(path) .await .map_err(serr)?; - let storage = Arc::new(RwLock::new(storage)) as Arc>; + let storage: Box = Box::new(storage); let mut idx = Self::new_with_storage(storage)?; idx.rebuild().await?; Ok(idx) @@ -419,7 +420,7 @@ impl GraphIndex { let storage = crate::storage::sqlite::SqliteStorage::open(path) .await .map_err(serr)?; - let storage = Arc::new(RwLock::new(storage)) as Arc>; + let storage: Box = Box::new(storage); let mut idx = Self::new_with_storage(storage)?; idx.rebuild().await?; Ok(idx) @@ -450,7 +451,7 @@ impl GraphIndex { crate::storage::redis::RedisStorage::new(client, &format!("codegraph:idx:{db}")) .await .map_err(serr)?; - let storage = Arc::new(RwLock::new(storage)) as Arc>; + let storage: Box = Box::new(storage); let mut idx = Self::new_with_storage(storage)?; idx.rebuild().await?; Ok(idx) @@ -493,8 +494,7 @@ impl GraphIndex { .ensure_registered(shard, route.root()) .await .map_err(serr)?; - let storage = Arc::new(RwLock::new(storage)) - as Arc>; + let storage: Box = Box::new(storage); let mut idx = Self::new_with_storage(storage)?; idx.rebuild().await?; Ok(idx) @@ -513,8 +513,7 @@ impl GraphIndex { .ensure_registered(shard, route.root()) .await .map_err(serr)?; - let storage = Arc::new(RwLock::new(storage)) - as Arc>; + let storage: Box = Box::new(storage); let mut idx = Self::new_with_storage(storage)?; idx.rebuild().await?; Ok(idx) @@ -531,11 +530,15 @@ impl GraphIndex { } } - fn new_with_storage(storage: Arc>) -> Result { + fn new_with_storage(storage: Box) -> Result { + // Wrap mọi backend bằng LRU read-cache để giảm gọi xuống storage + // (SQL/remote) cho các read path nóng (node/children/chain/entity). + const CACHE_CAPACITY: usize = 8192; + let storage = CachedStorage::wrap(storage, CACHE_CAPACITY); // Name engine luôn in-memory (như semgraph SearchIndex) — storage riêng // để record id (1..N) không đụng record của chain engine (func ids). - let name_storage = Arc::new(RwLock::new(InMemoryStorage::default())) - as Arc>; + let name_storage = + CachedStorage::wrap(Box::new(InMemoryStorage::default()), CACHE_CAPACITY); // Embedding chỉ bật khi config `[embedding].backend = "fastembed"` (opt-in). // Nếu bật mà model tải thất bại → lỗi rõ ràng (KHÔNG fallback silent). let (backend, enabled) = if embedding_enabled() { @@ -650,6 +653,19 @@ impl GraphIndex { self.rebuild_chain_engine(None).await?; self.rebuild_name_engine(None).await?; self.rebuild_vector_index(None).await?; + // Persist counts để `codegraph_status` đọc O(1) (không rebuild lại). + { + let mut st = self.storage.write().await; + st.set_stats(IndexCounts { + symbols: self.symbols.len() as u64, + chains: self.chains_map.len() as u64, + edges: self.edges.len() as u64, + files: self.files.len() as u64, + next_id: self.next_id, + }) + .await + .map_err(serr)?; + } Ok(()) } @@ -955,6 +971,15 @@ impl GraphIndex { let mut st = self.storage.write().await; st.save_next_id(self.next_id).await.map_err(serr)?; st.set_version(self.version).await.map_err(serr)?; + st.set_stats(IndexCounts { + symbols: self.symbols.len() as u64, + chains: self.chains_map.len() as u64, + edges: self.edges.len() as u64, + files: self.files.len() as u64, + next_id: self.next_id, + }) + .await + .map_err(serr)?; } Ok(()) } diff --git a/crates/codegraph-graph/src/lru.rs b/crates/codegraph-graph/src/lru.rs new file mode 100644 index 000000000..4a1447d67 --- /dev/null +++ b/crates/codegraph-graph/src/lru.rs @@ -0,0 +1,674 @@ +use dashmap::DashMap; +use parking_lot::Mutex; +use std::collections::hash_map::DefaultHasher; +use std::fmt; +use std::hash::{Hash, Hasher}; +use std::sync::Arc; +use std::sync::atomic::{AtomicUsize, Ordering}; + +const NULL: usize = usize::MAX; + +// --- CẤU TRÚC DỮ LIỆU --- + +struct Node { + key: Option, + value: Option, + next: AtomicUsize, + prev: AtomicUsize, +} + +struct HeadTail { + first: usize, + last: usize, +} + +/// AlignedShard giúp mỗi Mutex nằm riêng trên một Cache Line (64 bytes). +/// Điều này loại bỏ hiện tượng False Sharing, giúp tăng tốc ghi đa luồng. +#[repr(align(64))] +struct AlignedShard { + mutex: Mutex, +} + +pub struct LruCache { + mapping: DashMap, + caching: Box<[Node]>, + shards: [AlignedShard; S], + shard_mask: usize, + pub on_removing: Option>, + pub on_updating: Option>, +} + +impl fmt::Debug for LruCache +where + K: fmt::Debug + std::hash::Hash + Eq, + V: fmt::Debug, +{ + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.debug_struct("LruCache") + .field("mapping", &self.mapping) + .field("caching_len", &self.caching.len()) + .field("shard_mask", &self.shard_mask) + .field("on_removing", &self.on_removing.as_ref().map(|_| "Closure")) + .field("on_updating", &self.on_updating.as_ref().map(|_| "Closure")) + .finish() + } +} +// --- IMPLEMENTATION --- + +impl LruCache +where + K: Clone + Hash + Eq + Send + Sync, + V: Clone + Send + Sync, +{ + pub fn new(total_capacity: usize) -> Self { + // S phải là lũy thừa của 2 để dùng bitwise AND thay cho phép chia lấy dư (%) + assert!( + S > 0 && S.is_power_of_two(), + "SHARD_COUNT (S) phải là lũy thừa của 2 (ví dụ: 8, 16, 32)" + ); + + let capacity_per_shard = total_capacity.div_ceil(S); + let actual_total = capacity_per_shard * S; + + // 1. Khởi tạo Arena bộ nhớ phẳng + let mut caching_vec = Vec::with_capacity(actual_total); + for shard_idx in 0..S { + let offset = shard_idx * capacity_per_shard; + for i in 0..capacity_per_shard { + let current = offset + i; + caching_vec.push(Node { + key: None, + value: None, + next: AtomicUsize::new(if i + 1 < capacity_per_shard { + current + 1 + } else { + NULL + }), + prev: AtomicUsize::new(if i > 0 { current - 1 } else { NULL }), + }); + } + } + + // 2. Khởi tạo mảng các Shard Mutex (đã được aligned) + let shards = std::array::from_fn(|i| { + let offset = i * capacity_per_shard; + AlignedShard { + mutex: Mutex::new(HeadTail { + first: if capacity_per_shard > 0 { offset } else { NULL }, + last: if capacity_per_shard > 0 { + offset + capacity_per_shard - 1 + } else { + NULL + }, + }), + } + }); + + Self { + mapping: DashMap::with_capacity(actual_total), + caching: caching_vec.into_boxed_slice(), + shards, + shard_mask: S - 1, + on_removing: None, + on_updating: None, + } + } + + #[inline] + pub fn get_shard_idx(&self, key: &K) -> usize { + let mut s = DefaultHasher::new(); + key.hash(&mut s); + (s.finish() as usize) & self.shard_mask + } + + pub fn get(&self, key: &K) -> Option { + let index = *self.mapping.get(key)?; + + // Đọc giá trị an toàn (Node này chắc chắn tồn tại vì mapping đang giữ nó) + let val = self.caching[index].value.as_ref()?.clone(); + + // Optimistic LRU Update: Dùng try_lock để không làm chậm luồng Read + let shard_idx = self.get_shard_idx(key); + if let Some(mut ht) = self.shards[shard_idx].mutex.try_lock() { + self.move_to_front_inside_lock(&mut ht, index); + } + + Some(val) + } + + pub fn put(&self, key: K, value: V) { + let shard_idx = self.get_shard_idx(&key); + + // Case 1: Key đã tồn tại (Update) + if let Some(entry) = self.mapping.get_mut(&key) { + let index = *entry.value(); + if let Some(cb) = &self.on_updating { + cb(key.clone(), value.clone()); + } + + unsafe { + let node_ptr = &self.caching[index] as *const Node as *mut Node; + (*node_ptr).value = Some(value); + } + drop(entry); + + // Cập nhật thứ tự (Có thể dùng try_lock hoặc lock tùy độ ưu tiên) + if let Some(mut ht) = self.shards[shard_idx].mutex.try_lock() { + self.move_to_front_inside_lock(&mut ht, index); + } + return; + } + + // Case 2: Ghi mới (Bắt buộc dùng lock cứng để bảo vệ tính nhất quán) + let mut ht = self.shards[shard_idx].mutex.lock(); + let last_idx = ht.last; + if last_idx == NULL { + return; + } + + let node = &self.caching[last_idx]; + + // Đuổi dữ liệu cũ nếu có + if let Some(ref old_key) = node.key { + self.mapping.remove(old_key); + if let Some(cb) = &self.on_removing { + cb(old_key.clone(), node.value.as_ref().unwrap().clone()); + } + } + + // Ghi dữ liệu mới vào Node cuối của Shard + unsafe { + let node_ptr = node as *const Node as *mut Node; + (*node_ptr).key = Some(key.clone()); + (*node_ptr).value = Some(value); + } + + self.mapping.insert(key, last_idx); + self.move_to_front_inside_lock(&mut ht, last_idx); + } + + /// Xoá entry khỏi cache theo key + /// Chỉ remove khỏi DashMap, slot trong arena được tái sử dụng khi `put` overwrite. + pub fn remove(&self, key: &K) -> Option { + let (_, index) = self.mapping.remove(key)?; + self.caching[index].value.clone() + } + + /// Xoá toàn bộ entry (dùng khi invalidate hàng loạt, VD sau transaction + /// commit hoặc `clear_*` của storage). Reset cả arena lẫn linked-list. + pub fn clear(&self) { + self.mapping.clear(); + let cap_per_shard = self.caching.len() / S; + for shard_idx in 0..S { + let offset = shard_idx * cap_per_shard; + for i in 0..cap_per_shard { + let cur = offset + i; + unsafe { + let p = &self.caching[cur] as *const Node as *mut Node; + (*p).key = None; + (*p).value = None; + (*p).next.store( + if i + 1 < cap_per_shard { cur + 1 } else { NULL }, + Ordering::Release, + ); + (*p).prev + .store(if i > 0 { cur - 1 } else { NULL }, Ordering::Release); + } + } + let mut ht = self.shards[shard_idx].mutex.lock(); + ht.first = if cap_per_shard > 0 { offset } else { NULL }; + ht.last = if cap_per_shard > 0 { + offset + cap_per_shard - 1 + } else { + NULL + }; + } + } + + fn move_to_front_inside_lock(&self, ht: &mut HeadTail, index: usize) { + if ht.first == index || ht.first == NULL { + return; + } + + let node = &self.caching[index]; + let p = node.prev.load(Ordering::Acquire); + let n = node.next.load(Ordering::Acquire); + + // Cắt node ra khỏi vị trí hiện tại + if p != NULL { + self.caching[p].next.store(n, Ordering::Release); + } + if n != NULL { + self.caching[n].prev.store(p, Ordering::Release); + } + + if index == ht.last { + ht.last = p; + } + + // Đưa lên đầu danh sách của Shard + let old_first = ht.first; + node.next.store(old_first, Ordering::Release); + node.prev.store(NULL, Ordering::Release); + + if old_first != NULL { + self.caching[old_first].prev.store(index, Ordering::Release); + } + + ht.first = index; + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Arc; + use std::thread; + use std::time::Duration; + + const SHARD_COUNT: usize = 32; + + #[test] + fn test_lru_cache_sharded_logic() { + let capacity_per_shard = 2; + let cache = LruCache::::new(capacity_per_shard * SHARD_COUNT); + + // Tìm 3 key rơi vào cùng 1 shard để test logic eviction + let mut keys = Vec::new(); + for i in 0..1000 { + if cache.get_shard_idx(&i) == 0 { + keys.push(i); + if keys.len() == 3 { + break; + } + } + } + let (k1, k2, k3) = (keys[0], keys[1], keys[2]); + + cache.put(k1, 10); + cache.put(k2, 20); + + assert_eq!(cache.get(&k1), Some(10)); // k1 lên head của shard + cache.put(k3, 30); // shard full (2 slot), evict k2 (vì k1 vừa được access) + + assert_eq!(cache.get(&k2), None); // k2 bị đuổi + assert_eq!(cache.get(&k1), Some(10)); + assert_eq!(cache.get(&k3), Some(30)); + } + + #[test] + fn test_update_existing_key() { + let cache = LruCache::::new(16 * 2); // 2 slot mỗi shard + cache.put(1, 10); + cache.put(1, 20); + + assert_eq!(cache.get(&1), Some(20)); + assert_eq!(cache.mapping.len(), 1); + + let index = *cache.mapping.get(&1).unwrap(); + cache.put(1, 30); + assert_eq!(index, *cache.mapping.get(&1).unwrap(), "Index không đổi"); + } + + #[test] + fn test_empty_cache() { + let cache = LruCache::::new(0); + cache.put(1, 10); + assert_eq!(cache.get(&1), None); + } + + #[test] + fn test_extreme_data_integrity() { + use std::collections::hash_map::DefaultHasher; + use std::hash::{Hash, Hasher}; + + let capacity_per_shard = 50; + let total_capacity = capacity_per_shard * SHARD_COUNT; + let cache = LruCache::::new(total_capacity); + + // Hàm tạo giá trị "chuẩn" theo Key để kiểm tra integrity + let gen_value = |k: usize| -> usize { + let mut s = DefaultHasher::new(); + k.hash(&mut s); + s.finish() as usize + }; + + let num_threads = 12; + let ops_per_thread = 2000; + + // --- PHASE 1: STRESS WRITE --- + thread::scope(|s| { + for t in 0..num_threads { + let cache_ref = &cache; + s.spawn(move || { + for i in 0..ops_per_thread { + let key = t * ops_per_thread + i; + let val = gen_value(key); + cache_ref.put(key, val); + } + }); + } + }); + + // --- PHASE 2: INTEGRITY VALIDATION --- + + // 1. Kiểm tra từng cặp Key-Value trong Mapping + for entry in cache.mapping.iter() { + let key = *entry.key(); + let index = *entry.value(); + + let node = &cache.caching[index]; + let stored_key = node.key.expect("Node trong mapping phải có key"); + let stored_val = node.value.expect("Node trong mapping phải có value"); + + assert_eq!( + key, stored_key, + "Data Corruption: Key trong mapping ({}) khác Key trong Node ({})", + key, stored_key + ); + assert_eq!( + stored_val, + gen_value(key), + "Data Corruption: Value của key {} bị sai lệch!", + key + ); + + // 2. Kiểm tra Shard Consistency: Key phải nằm đúng Shard của nó + let expected_shard = cache.get_shard_idx(&key); + // Kiểm tra xem index này có nằm trong dải bộ nhớ của Shard đó không + let actual_shard = index / capacity_per_shard; + assert_eq!( + expected_shard, actual_shard, + "Key {} nằm sai phân vùng Shard!", + key + ); + } + + // 3. Kiểm tra tính toàn vẹn của cấu trúc Danh sách liên kết (Double-ended check) + for s_idx in 0..SHARD_COUNT { + let ht = cache.shards[s_idx].mutex.lock(); + let mut forward_count = 0; + let mut backward_count = 0; + + // Duyệt xuôi: Head -> Tail + let mut curr = ht.first; + let mut last_seen = NULL; + while curr != NULL { + forward_count += 1; + last_seen = curr; + curr = cache.caching[curr].next.load(Ordering::Acquire); + } + assert_eq!( + last_seen, ht.last, + "Tail của Shard {} không khớp khi duyệt xuôi", + s_idx + ); + + // Duyệt ngược: Tail -> Head + let mut curr = ht.last; + let mut first_seen = NULL; + while curr != NULL { + backward_count += 1; + first_seen = curr; + curr = cache.caching[curr].prev.load(Ordering::Acquire); + } + assert_eq!( + first_seen, ht.first, + "Head của Shard {} không khớp khi duyệt ngược", + s_idx + ); + assert_eq!( + forward_count, backward_count, + "Số lượng node duyệt xuôi và ngược không bằng nhau ở Shard {}", + s_idx + ); + assert_eq!( + forward_count, capacity_per_shard, + "Shard {} không đủ số lượng node", + s_idx + ); + } + + println!("🚀 [PASSED] Dữ liệu chuẩn 100%, không phát hiện Race Condition trên Node!"); + } + + #[test] + fn test_internal_state_after_eviction_sharded() { + // Để dễ test eviction, ta chọn capacity sao cho mỗi shard có đúng 2 slot + let capacity_per_shard = 2; + let total_capacity = capacity_per_shard * SHARD_COUNT; + let cache = LruCache::::new(total_capacity); + + // 1. Tìm 3 key sao cho chúng rơi vào CÙNG MỘT SHARD + // Điều này quan trọng vì mỗi shard tự quản lý việc đuổi (eviction) riêng + let mut keys = Vec::new(); + + for i in 0..1000 { + if cache.get_shard_idx(&i) == 0 { + keys.push(i); + if keys.len() == 3 { + break; + } + } + } + + let k1 = keys[0]; + let k2 = keys[1]; + let k3 = keys[2]; + + // Giai đoạn lấp đầy 2 slot của Shard 0 + cache.put(k1, 10); + cache.put(k2, 20); + + // Lấy index của k1 trước khi nó bị đuổi + let index_of_k1 = *cache.mapping.get(&k1).expect("Key 1 phải tồn tại").value(); + + // 2. Evict k1 bằng cách chèn k3 (vào cùng shard 0) + cache.put(k3, 30); + + // Kiểm tra mapping + assert_eq!( + cache.mapping.get(&k3).map(|e| *e.value()), + Some(index_of_k1), + "Key 3 phải chiếm slot của Key 1" + ); + assert!(cache.mapping.get(&k1).is_none(), "Key 1 phải bị đuổi"); + + // 3. Lock đúng Shard 0 để kiểm tra Head/Tail + let shard_idx = cache.get_shard_idx(&k3); + let ht = cache.shards[shard_idx].mutex.lock(); + + let mru_index = *cache.mapping.get(&k3).unwrap().value(); + let lru_index = *cache.mapping.get(&k2).unwrap().value(); + + assert_eq!(ht.first, mru_index, "Key 3 phải là đầu danh sách của shard"); + assert_eq!(ht.last, lru_index, "Key 2 phải là cuối danh sách của shard"); + + // 4. Kiểm tra liên kết giữa các node trong Arena + let mru_node = &cache.caching[mru_index]; + let lru_node = &cache.caching[lru_index]; + + assert_eq!(mru_node.key, Some(k3)); + assert_eq!(mru_node.next.load(Ordering::Relaxed), lru_index); + assert_eq!(mru_node.prev.load(Ordering::Relaxed), NULL); + + assert_eq!(lru_node.key, Some(k2)); + assert_eq!(lru_node.next.load(Ordering::Relaxed), NULL); + assert_eq!(lru_node.prev.load(Ordering::Relaxed), mru_index); + } + + #[test] + fn test_lru_deadlock() { + // Khởi tạo cache với capacity 10 + let cache = Arc::new(LruCache::::new(16)); + + // Giả lập dữ liệu ban đầu + cache.put(1, "A".to_string()); + cache.put(2, "B".to_string()); + + let cache_clone1 = Arc::clone(&cache); + let t1 = thread::spawn(move || { + for _ in 0..1000 { + // Thread 1: Liên tục gọi put (chiếm nhiều lock bên trong) + cache_clone1.put(1, "A_updated".to_string()); + } + }); + + let cache_clone2 = Arc::clone(&cache); + let t2 = thread::spawn(move || { + for _ in 0..1000 { + // Thread 2: Liên tục gọi get (cũng gây move_to_front và chiếm lock) + cache_clone2.get(&2); + } + }); + + // Đợi 5 giây. Nếu code đúng O(1) thì 2000 thao tác này phải xong trong < 1s. + // Nếu sau 5s không xong nghĩa là đã Deadlock. + let result = thread::spawn(move || { + t1.join().unwrap(); + t2.join().unwrap(); + }); + + // Cơ chế check timeout cho test + if wait_timeout(result, Duration::from_secs(5)).is_err() { + panic!( + "TEST FAILED: Deadlock detected! Cấu trúc nhiều RwLock lồng nhau đã làm treo thread." + ); + } + } + + fn wait_timeout( + handle: thread::JoinHandle, + timeout: Duration, + ) -> Result<(), ()> { + let (tx, rx) = std::sync::mpsc::channel(); + thread::spawn(move || { + let _ = handle.join(); + let _ = tx.send(()); + }); + // Đợi kết quả từ thread trong khoảng timeout + rx.recv_timeout(timeout).map_err(|_| ()) + } + + #[test] + fn prove_deadlock_extremes() { + use std::sync::Arc; + use std::thread; + use std::time::Duration; + + let cache = Arc::new(LruCache::::new(100)); + + // Nạp sẵn dữ liệu để thread 2 luôn rơi vào nhánh move_to_front + for i in 0..100 { + cache.put(i, i); + } + + let cache_clone = cache.clone(); + let t1 = thread::spawn(move || { + for i in 100..10000 { + // Thread 1: Liên tục PUT key mới (gây áp lực lên chèn node và cập nhật first/last) + cache_clone.put(i, i); + } + }); + + let cache_clone2 = cache.clone(); + let t2 = thread::spawn(move || { + for _ in 0..10000 { + // Thread 2: Liên tục GET key cũ (gây áp lực lên move_to_front) + // move_to_front sẽ chiếm caching.write rồi lại đòi first.write/read + cache_clone2.get(&50); + } + }); + + // Nếu không treo, 20.000 ops này phải xong trong < 1 giây + let (tx, rx) = std::sync::mpsc::channel(); + thread::spawn(move || { + t1.join().unwrap(); + t2.join().unwrap(); + let _ = tx.send(()); + }); + + if rx.recv_timeout(Duration::from_secs(10)).is_err() { + panic!("DEADLOCK CONFIRMED: Hệ thống đã treo hoàn toàn sau 10 giây!"); + } + } + + #[test] + fn test_no_data_loss_and_leak() { + use std::sync::atomic::{AtomicUsize, Ordering}; + + let capacity_per_shard = 100; + let total_capacity = capacity_per_shard * SHARD_COUNT; + let evicted_count = Arc::new(AtomicUsize::new(0)); + + // Setup cache với callback đếm số lần bị đuổi + let evicted_clone = Arc::clone(&evicted_count); + let mut cache = LruCache::::new(total_capacity); + cache.on_removing = Some(Arc::new(move |_, _| { + evicted_clone.fetch_add(1, Ordering::SeqCst); + })); + + let num_threads = 8; + let ops_per_thread = 5000; + let total_ops = num_threads * ops_per_thread; + + thread::scope(|s| { + for t in 0..num_threads { + let cache_ref = &cache; + s.spawn(move || { + for i in 0..ops_per_thread { + let key = t * ops_per_thread + i; + cache_ref.put(key, i); + } + }); + } + }); + + // --- BẮT ĐẦU VALIDATION --- + + // 1. Kiểm tra Mapping size + // Số lượng phần tử hiện tại phải bằng total_capacity vì chúng ta chèn vượt ngưỡng rất nhiều + assert_eq!( + cache.mapping.len(), + total_capacity, + "Mapping phải đầy khít capacity" + ); + + // 2. Kiểm tra tính nhất quán của Linked List (Duyệt từng Shard) + let mut total_nodes_in_lists = 0; + for i in 0..SHARD_COUNT { + let ht = cache.shards[i].mutex.lock(); + let mut count = 0; + let mut curr = ht.first; + let mut visited = std::collections::HashSet::new(); + + while curr != NULL { + assert!( + visited.insert(curr), + "Phát hiện chu trình (vòng lặp vô tận) trong Shard {}", + i + ); + count += 1; + curr = cache.caching[curr].next.load(Ordering::Acquire); + } + assert_eq!( + count, capacity_per_shard, + "Shard {} bị thiếu node trong danh sách liên kết", + i + ); + total_nodes_in_lists += count; + } + assert_eq!(total_nodes_in_lists, total_capacity); + + // 3. Kiểm tra số lượng đã bị đuổi (Eviction Balance) + // Công thức: Tổng Put - Capacity = Số lần phải Evict + let actual_evicted = evicted_count.load(Ordering::SeqCst); + let expected_evicted = total_ops - total_capacity; + assert_eq!( + actual_evicted, expected_evicted, + "Số lượng callback xóa không khớp với logic eviction" + ); + + println!("✅ Test passed: Không có dữ liệu bị 'lạc trôi', Linked List hoàn hảo!"); + } +} diff --git a/crates/codegraph-graph/src/shared.rs b/crates/codegraph-graph/src/shared.rs index 7eb9cbfd4..2ace65b68 100644 --- a/crates/codegraph-graph/src/shared.rs +++ b/crates/codegraph-graph/src/shared.rs @@ -15,7 +15,8 @@ //! được chọn theo scheme trong route, không phải theo thứ tự feature. use crate::GraphIndex; -use codegraph_core::{Result, StorageRoute}; +use crate::storage::Storage; +use codegraph_core::{Result, SemgraphStats, StorageRoute}; use std::sync::Arc; use tokio::sync::{Mutex, RwLock}; @@ -38,6 +39,8 @@ pub struct SharedGraphIndex { state: RwLock, /// Serialize rebuild — N request stale đồng thời chỉ 1 lần rebuild. rebuild_lock: Arc>, + /// Storage read-only cache để `stats_cached` đọc counts O(1) không rebuild. + stats_storage: RwLock>>, } impl SharedGraphIndex { @@ -58,6 +61,7 @@ impl SharedGraphIndex { ready: false, }), rebuild_lock: Arc::new(Mutex::new(())), + stats_storage: RwLock::new(None), }) } @@ -201,6 +205,91 @@ impl SharedGraphIndex { state.ready = true; Ok(()) } + + /// Đọc counts tổng hợp từ đĩa (`sg_stats`) mà KHÔNG rebuild in-memory + /// `GraphIndex` — O(1) với repo lớn. Hỗ trợ sqlite/lmdb/postgres/mysql; + /// backend khác / index cũ thiếu bảng → trả `None` để caller fallback rebuild. + /// + /// Trả `None` cả khi counts toàn 0 (index cũ chưa ghi `sg_stats`) để không + /// trình ra số 0 sai lệch. + pub async fn stats_cached(&self) -> Option { + let storage = self.stats_storage_handle().await?; + let counts = storage.stats().await.ok()?; + if counts.symbols == 0 && counts.chains == 0 && counts.edges == 0 && counts.files == 0 { + return None; + } + Some(SemgraphStats { + symbols: counts.symbols, + chains: counts.chains, + edges: counts.edges, + files: counts.files, + next_id: counts.next_id, + }) + } + + /// Lấy (và cache) storage read-only từ route để đọc `sg_stats` không rebuild. + /// Hỗ trợ: sqlite (Local), lmdb (Local, read-only để không tranh lock), + /// postgres/mysql (Sharded — resolve dsn đầu + repo_id). Backend khác + /// (redis/unknown) → `None` → caller fallback rebuild. + async fn stats_storage_handle(&self) -> Option> { + { + let g = self.stats_storage.read().await; + if let Some(s) = g.as_ref() { + return Some(s.clone()); + } + } + let st: Arc = match &self.route { + #[cfg(feature = "sqlite")] + Some(StorageRoute::Local(d)) if d.starts_with("sqlite://") => { + let s = crate::storage::sqlite::SqliteStorage::open(trim_scheme(d)) + .await + .ok()?; + Arc::new(s) + } + #[cfg(feature = "lmdb")] + Some(StorageRoute::Local(d)) if d.starts_with("lmdb://") => { + let s = crate::storage::lmdb::LmdbStorage::open(trim_scheme(d)) + .await + .ok()?; + Arc::new(s) + } + #[cfg(any(feature = "postgres", feature = "mysql"))] + Some(StorageRoute::Sharded { dsns, repo_id, .. }) => { + let dsn = dsns.first()?; + let rid = (*repo_id)?; + if dsn.starts_with("postgres://") { + #[cfg(feature = "postgres")] + { + let s = crate::storage::postgres::PostgresStorage::open(dsn, rid) + .await + .ok()?; + Arc::new(s) + } + #[cfg(not(feature = "postgres"))] + { + return None; + } + } else if dsn.starts_with("mysql://") { + #[cfg(feature = "mysql")] + { + let s = crate::storage::mysql::MySqlStorage::open(dsn, rid) + .await + .ok()?; + Arc::new(s) + } + #[cfg(not(feature = "mysql"))] + { + return None; + } + } else { + return None; + } + } + _ => return None, + }; + *self.stats_storage.write().await = Some(st.clone()); + Some(st) + } } /// Bỏ `scheme://` khỏi DSN — trả phần còn lại (path cho probe file). @@ -300,4 +389,34 @@ mod tests { assert_eq!(idx2.stats().symbols, 1); assert_eq!(idx2.symbol_by_id(SYMBOL_BASE).unwrap().name, "x"); } + + /// `stats_cached` đọc counts từ đĩa (`sg_stats`) mà KHÔNG rebuild in-memory + /// `GraphIndex` — xác nhận `codegraph_status` tức thì trên repo lớn. + #[cfg(feature = "sqlite")] + #[tokio::test] + async fn sqlite_stats_cached_reads_disk_without_rebuild() { + let dir = tempfile::tempdir().unwrap(); + let db_path = dir.path().join("db.sqlite"); + let db_str = format!("sqlite://{}", db_path.to_string_lossy()); + + // Index qua process riêng — ghi `sg_stats` lúc ingest. + { + let mut idx = GraphIndex::open(&db_str).await.unwrap(); + let r = mk_result( + "a.ts", + vec![sym("a", SYMBOL_BASE), sym("b", SYMBOL_BASE + 1)], + vec![SYMBOL_BASE, SYMBOL_BASE + 1], + ); + idx.ingest(&[r]).await.unwrap(); + } + + let sgi = SharedGraphIndex::open(Some(db_str.clone())).await.unwrap(); + // Chưa gọi `ensure_fresh` — `stats_cached` mở storage riêng đọc `sg_stats`. + let stats = sgi.stats_cached().await.expect("sg_stats đã populate"); + assert_eq!(stats.symbols, 2); + assert_eq!(stats.chains, 1); + assert_eq!(stats.edges, 1); + assert_eq!(stats.files, 1); + assert_eq!(stats.next_id, SYMBOL_BASE + 2); + } } diff --git a/crates/codegraph-graph/src/storage.rs b/crates/codegraph-graph/src/storage.rs index 13ed55b16..878d16140 100644 --- a/crates/codegraph-graph/src/storage.rs +++ b/crates/codegraph-graph/src/storage.rs @@ -16,6 +16,9 @@ use std::sync::{Arc, RwLock}; use async_trait::async_trait; use codegraph_core::{FileInfo, Symbol}; +/// Decorator `Storage` bọc LRU cache (giảm gọi xuống backend). +pub mod cached; + #[cfg(feature = "sqlite")] pub mod sqlite; @@ -141,6 +144,17 @@ pub trait Tx: Send { // ==================== Storage trait ==================== +/// Counts tổng hợp của index — `codegraph_status` đọc O(1) từ đĩa mà không +/// cần rebuild in-memory `GraphIndex` (vốn rất đắt trên repo lớn). +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)] +pub struct IndexCounts { + pub symbols: u64, + pub chains: u64, + pub edges: u64, + pub files: u64, + pub next_id: u64, +} + /// Radix-node storage: node management + transaction. #[async_trait] pub trait Storage: Send + Sync { @@ -343,6 +357,16 @@ pub trait Storage: Send + Sync { async fn set_version(&mut self, _v: u64) -> Result<()> { Ok(()) } + /// Lưu counts tổng hợp (symbols/chains/edges/files) — `codegraph_status` + /// đọc trực tiếp từ đĩa, bỏ qua rebuild in-memory. Mặc định: no-op. + async fn set_stats(&mut self, _s: IndexCounts) -> Result<()> { + Ok(()) + } + /// Đọc counts tổng hợp từ đĩa. Mặc định: `Ok(IndexCounts::default())` + /// (toàn 0). Backend không lưu → caller fallback sang rebuild. + async fn stats(&self) -> Result { + Ok(IndexCounts::default()) + } /// Xoá toàn bộ entity data (symbols/next_id/call_records/call_names/files/ /// version) — dùng khi full re-index. Mặc định: no-op. async fn clear_entities(&mut self) -> Result<()> { diff --git a/crates/codegraph-graph/src/storage/cached.rs b/crates/codegraph-graph/src/storage/cached.rs new file mode 100644 index 000000000..654eaf2fa --- /dev/null +++ b/crates/codegraph-graph/src/storage/cached.rs @@ -0,0 +1,623 @@ +//! `CachedStorage` — decorator bọc một `Storage` bất kỳ bằng `LruCache` sharded +//! để giảm số lần gọi xuống backend (SQL/remote) cho các read path nóng. +//! +//! - Các `get_*` nóng (node/children/chain/meta/edge/symbol/embedding/...) đọc +//! cache trước; miss → gọi inner → populate. +//! - Các method ghi (`set_*`/`new_node`/`update_node`/`set_root`/...) ghi qua +//! inner VÀ invalidate đúng cache liên quan. +//! - Transaction: `new_tx` trả `CachedTx`; khi `commit` xong sẽ `clear_radix()` +//! (node/children/roots/shortcuts) vì tx chỉ sửa cấu trúc radix — entity cache +//! (symbol/embedding/call) giữ nguyên, không bị lạnh. +//! +//! Decorator này trong suốt: mọi backend (InMemory/Sqlite/Lmdb/Redis/RDBMS) +//! đều dùng được, behaviour đúng bằng inner (chỉ thêm lớp cache). + +use std::collections::HashMap; +use std::sync::Arc; + +use async_trait::async_trait; +use codegraph_core::{FileInfo, Symbol}; + +use crate::lru::LruCache; +use crate::storage::{IndexCounts, Storage, StorageError, Tx}; + +/// Số shard của mỗi `LruCache` — phải lũy thừa của 2. +const SHARDS: usize = 32; + +/// Tập hợp các cache theo từng loại read method. Dùng `Arc` để `CachedTx` +/// (được tạo từ `new_tx`) cũng giữ được tham chiếu tới cùng bộ cache để +/// invalidate khi commit. +struct CacheSet { + nodes: LruCache, usize), SHARDS>, + children: LruCache, SHARDS>, + chains: LruCache, SHARDS>, + metas: LruCache, SHARDS>, + key_lens: LruCache, + edge_data: LruCache, SHARDS>, + node_meta: LruCache, SHARDS>, + roots: LruCache, + shortcuts: LruCache<(usize, Vec), Vec, SHARDS>, + symbols: LruCache, + embeddings: LruCache, SHARDS>, + call_records: LruCache, SHARDS>, + call_name_index: LruCache, SHARDS>, +} + +impl CacheSet { + fn new(capacity: usize) -> Self { + Self { + nodes: LruCache::new(capacity), + children: LruCache::new(capacity), + chains: LruCache::new(capacity), + metas: LruCache::new(capacity), + key_lens: LruCache::new(capacity), + edge_data: LruCache::new(capacity), + node_meta: LruCache::new(capacity), + roots: LruCache::new(capacity), + shortcuts: LruCache::new(capacity), + symbols: LruCache::new(capacity), + embeddings: LruCache::new(capacity), + call_records: LruCache::new(capacity), + call_name_index: LruCache::new(capacity), + } + } + + /// Invalidate mọi cache liên quan đến cấu trúc radix (chỉ những thứ tx sửa). + fn clear_radix(&self) { + self.nodes.clear(); + self.children.clear(); + self.roots.clear(); + self.shortcuts.clear(); + } + + /// Invalidate toàn bộ (dùng cho `clear_entities` / reset lớn). + #[allow(dead_code)] + fn clear_all(&self) { + self.clear_radix(); + self.chains.clear(); + self.metas.clear(); + self.key_lens.clear(); + self.edge_data.clear(); + self.node_meta.clear(); + self.symbols.clear(); + self.embeddings.clear(); + self.call_records.clear(); + self.call_name_index.clear(); + } +} + +/// Decorator `Storage` có LRU cache. `inner` là `Box` — backend tự +/// quản lý concurrency của nó, decorator không cần lock riêng. +pub struct CachedStorage { + inner: Box, + caches: Arc, +} + +impl CachedStorage { + /// Bọc một `Storage` bất kỳ. Trả về `Arc>` để có thể + /// truyền thẳng vào `GraphIndex` (cùng kiểu với backend gốc). + pub fn wrap(inner: Box, capacity: usize) -> Arc> { + Arc::new(tokio::sync::RwLock::new(CachedStorage { + inner, + caches: Arc::new(CacheSet::new(capacity)), + })) + } +} + +#[async_trait] +impl Storage for CachedStorage { + // ── Node management (cached) ── + async fn new_node(&mut self, prefix: Vec, record: usize) -> Result { + let id = self.inner.new_node(prefix, record).await?; + self.caches.nodes.remove(&id); + Ok(id) + } + + async fn update_node( + &mut self, + id: usize, + prefix: Option>, + record: Option, + ) -> Result<(), StorageError> { + self.inner.update_node(id, prefix, record).await?; + self.caches.nodes.remove(&id); + Ok(()) + } + + async fn get_node(&self, id: usize) -> Result<(Vec, usize), StorageError> { + if let Some(v) = self.caches.nodes.get(&id) { + return Ok(v); + } + let v = self.inner.get_node(id).await?; + self.caches.nodes.put(id, v.clone()); + Ok(v) + } + + async fn get_children(&self, id: usize) -> Result, StorageError> { + if let Some(v) = self.caches.children.get(&id) { + return Ok(v); + } + let v = self.inner.get_children(id).await?; + self.caches.children.put(id, v.clone()); + Ok(v) + } + + // ── Bloom (không cache — dùng prune nhánh, sai = search sai) ── + #[cfg(feature = "bloom-search")] + async fn set_node_bloom(&mut self, id: usize, bloom: &[u8]) -> Result<(), StorageError> { + self.inner.set_node_bloom(id, bloom).await + } + + #[cfg(feature = "bloom-search")] + async fn get_node_bloom(&self, id: usize) -> Result>, StorageError> { + self.inner.get_node_bloom(id).await + } + + // ── Edge data (cached) ── + async fn set_edge_data(&mut self, edge: usize, data: &[u8]) -> Result<(), StorageError> { + self.inner.set_edge_data(edge, data).await?; + self.caches.edge_data.remove(&edge); + Ok(()) + } + + async fn get_edge_data(&self, edge: usize) -> Result>, StorageError> { + if let Some(v) = self.caches.edge_data.get(&edge) { + return Ok(Some(v)); + } + let v = self.inner.get_edge_data(edge).await?; + if let Some(ref b) = v { + self.caches.edge_data.put(edge, b.clone()); + } + Ok(v) + } + + async fn clear_edges(&mut self) -> Result<(), StorageError> { + self.inner.clear_edges().await?; + self.caches.edge_data.clear(); + Ok(()) + } + + async fn for_each_edge_data( + &self, + f: &mut (dyn for<'a> FnMut(usize, &'a [u8]) -> Result<(), StorageError> + Send), + ) -> Result<(), StorageError> { + self.inner.for_each_edge_data(f).await + } + + // ── Node metadata (cached) ── + async fn set_node_meta(&mut self, elem: usize, meta: &[u8]) -> Result<(), StorageError> { + self.inner.set_node_meta(elem, meta).await?; + self.caches.node_meta.remove(&elem); + Ok(()) + } + + async fn get_node_meta(&self, elem: usize) -> Result>, StorageError> { + if let Some(v) = self.caches.node_meta.get(&elem) { + return Ok(Some(v)); + } + let v = self.inner.get_node_meta(elem).await?; + if let Some(ref b) = v { + self.caches.node_meta.put(elem, b.clone()); + } + Ok(v) + } + + async fn clear_node_meta(&mut self) -> Result<(), StorageError> { + self.inner.clear_node_meta().await?; + self.caches.node_meta.clear(); + Ok(()) + } + + // ── Chain (cached) ── + async fn set_chain(&mut self, record: usize, chain: &[u64]) -> Result<(), StorageError> { + self.inner.set_chain(record, chain).await?; + self.caches.chains.remove(&record); + Ok(()) + } + + async fn get_chain(&self, record: usize) -> Result>, StorageError> { + if let Some(v) = self.caches.chains.get(&record) { + return Ok(Some(v)); + } + let v = self.inner.get_chain(record).await?; + if let Some(ref c) = v { + self.caches.chains.put(record, c.clone()); + } + Ok(v) + } + + async fn clear_chains(&mut self) -> Result<(), StorageError> { + self.inner.clear_chains().await?; + self.caches.chains.clear(); + Ok(()) + } + + // ── Shard roots (cached) ── + async fn set_root(&mut self, shard: usize, root: usize) -> Result<(), StorageError> { + self.inner.set_root(shard, root).await?; + self.caches.roots.remove(&shard); + Ok(()) + } + + async fn get_root(&self, shard: usize) -> Result { + if let Some(v) = self.caches.roots.get(&shard) { + return Ok(v); + } + let v = self.inner.get_root(shard).await?; + self.caches.roots.put(shard, v); + Ok(v) + } + + // ── Meta / key_len (cached) ── + async fn set_meta(&mut self, record: usize, meta: &[u8]) -> Result<(), StorageError> { + self.inner.set_meta(record, meta).await?; + self.caches.metas.remove(&record); + Ok(()) + } + + async fn get_meta(&self, record: usize) -> Result>, StorageError> { + if let Some(v) = self.caches.metas.get(&record) { + return Ok(Some(v)); + } + let v = self.inner.get_meta(record).await?; + if let Some(ref b) = v { + self.caches.metas.put(record, b.clone()); + } + Ok(v) + } + + async fn set_key_len(&mut self, record: usize, len: usize) -> Result<(), StorageError> { + self.inner.set_key_len(record, len).await?; + self.caches.key_lens.remove(&record); + Ok(()) + } + + async fn get_key_len(&self, record: usize) -> Result, StorageError> { + if let Some(v) = self.caches.key_lens.get(&record) { + return Ok(Some(v)); + } + let v = self.inner.get_key_len(record).await?; + if let Some(l) = v { + self.caches.key_lens.put(record, l); + } + Ok(v) + } + + // ── Shortcuts (cached) ── + async fn add_shortcut_node( + &mut self, + shard: usize, + elem: &[u8], + node_id: usize, + ) -> Result<(), StorageError> { + self.inner.add_shortcut_node(shard, elem, node_id).await?; + self.caches.shortcuts.remove(&(shard, elem.to_vec())); + Ok(()) + } + + async fn get_shortcut_nodes( + &self, + shard: usize, + elem: &[u8], + ) -> Result, StorageError> { + let key = (shard, elem.to_vec()); + if let Some(v) = self.caches.shortcuts.get(&key) { + return Ok(v); + } + let v = self.inner.get_shortcut_nodes(shard, elem).await?; + self.caches.shortcuts.put(key, v.clone()); + Ok(v) + } + + async fn clear_shortcuts(&mut self) -> Result<(), StorageError> { + self.inner.clear_shortcuts().await?; + self.caches.shortcuts.clear(); + Ok(()) + } + + // ── Entity store (symbols / calls / embeddings) ── + async fn save_symbol(&mut self, sym: &Symbol) -> Result<(), StorageError> { + self.inner.save_symbol(sym).await?; + self.caches.symbols.remove(&sym.id); + Ok(()) + } + + async fn load_symbol(&self, id: u64) -> Result, StorageError> { + if let Some(v) = self.caches.symbols.get(&id) { + return Ok(Some(v)); + } + let v = self.inner.load_symbol(id).await?; + if let Some(ref s) = v { + self.caches.symbols.put(id, s.clone()); + } + Ok(v) + } + + async fn load_all_symbols(&self) -> Result, StorageError> { + self.inner.load_all_symbols().await + } + + async fn save_next_id(&mut self, next: u64) -> Result<(), StorageError> { + self.inner.save_next_id(next).await + } + + async fn load_next_id(&self) -> Result { + self.inner.load_next_id().await + } + + async fn all_chains(&self) -> Result)>, StorageError> { + self.inner.all_chains().await + } + + async fn set_call_records(&mut self, func: u64, records: &[u8]) -> Result<(), StorageError> { + self.inner.set_call_records(func, records).await?; + self.caches.call_records.remove(&func); + Ok(()) + } + + async fn get_call_records(&self, func: u64) -> Result>, StorageError> { + if let Some(v) = self.caches.call_records.get(&func) { + return Ok(Some(v)); + } + let v = self.inner.get_call_records(func).await?; + if let Some(ref b) = v { + self.caches.call_records.put(func, b.clone()); + } + Ok(v) + } + + async fn all_call_records(&self) -> Result)>, StorageError> { + self.inner.all_call_records().await + } + + async fn set_call_name_index(&mut self, name: &str, sites: &[u8]) -> Result<(), StorageError> { + self.inner.set_call_name_index(name, sites).await?; + self.caches.call_name_index.remove(&name.to_string()); + Ok(()) + } + + async fn load_call_name_index(&self, name: &str) -> Result>, StorageError> { + if let Some(v) = self.caches.call_name_index.get(&name.to_string()) { + return Ok(Some(v)); + } + let v = self.inner.load_call_name_index(name).await?; + if let Some(ref b) = v { + self.caches.call_name_index.put(name.to_string(), b.clone()); + } + Ok(v) + } + + async fn all_call_name_indexes(&self) -> Result)>, StorageError> { + self.inner.all_call_name_indexes().await + } + + async fn upsert_file(&mut self, f: &FileInfo) -> Result<(), StorageError> { + self.inner.upsert_file(f).await + } + + async fn load_all_files(&self) -> Result, StorageError> { + self.inner.load_all_files().await + } + + async fn version(&self) -> Result { + self.inner.version().await + } + + async fn set_version(&mut self, v: u64) -> Result<(), StorageError> { + self.inner.set_version(v).await + } + + async fn set_stats(&mut self, s: IndexCounts) -> Result<(), StorageError> { + self.inner.set_stats(s).await + } + + async fn stats(&self) -> Result { + self.inner.stats().await + } + + async fn clear_entities(&mut self) -> Result<(), StorageError> { + self.inner.clear_entities().await?; + self.caches.clear_all(); + Ok(()) + } + + // ── Embeddings (cached) ── + async fn save_embedding(&mut self, symbol_id: u64, vector: &[f32]) -> Result<(), StorageError> { + self.inner.save_embedding(symbol_id, vector).await?; + self.caches.embeddings.remove(&symbol_id); + Ok(()) + } + + async fn load_embedding(&self, symbol_id: u64) -> Result>, StorageError> { + if let Some(v) = self.caches.embeddings.get(&symbol_id) { + return Ok(Some(v)); + } + let v = self.inner.load_embedding(symbol_id).await?; + if let Some(ref vec) = v { + self.caches.embeddings.put(symbol_id, vec.clone()); + } + Ok(v) + } + + async fn load_all_embeddings(&self) -> Result>, StorageError> { + self.inner.load_all_embeddings().await + } + + async fn clear_embeddings(&mut self) -> Result<(), StorageError> { + self.inner.clear_embeddings().await?; + self.caches.embeddings.clear(); + Ok(()) + } + + async fn knn( + &self, + query_vec: &[f32], + k: usize, + ) -> Result>, StorageError> { + self.inner.knn(query_vec, k).await + } + + // ── Transaction: wrap để invalidate radix cache khi commit ── + fn new_tx(&self) -> Box { + Box::new(CachedTx { + inner: self.inner.new_tx(), + caches: self.caches.clone(), + }) + } +} + +/// Tx bọc: delegate mọi mutation, khi `commit` xong thì `clear_radix()`. +struct CachedTx { + inner: Box, + caches: Arc, +} + +#[async_trait] +impl Tx for CachedTx { + async fn new_node(&mut self, prefix: Vec, record: usize) -> Result { + self.inner.new_node(prefix, record).await + } + + async fn update_node( + &mut self, + id: usize, + prefix: Option>, + record: Option, + ) -> Result<(), StorageError> { + self.inner.update_node(id, prefix, record).await + } + + async fn add_child(&mut self, parent: usize, child: usize) -> Result<(), StorageError> { + self.inner.add_child(parent, child).await + } + + async fn move_child( + &mut self, + from: usize, + to: usize, + child: usize, + ) -> Result<(), StorageError> { + self.inner.move_child(from, to, child).await + } + + async fn commit(self: Box) -> Result<(), StorageError> { + let CachedTx { inner, caches } = *self; + let res = inner.commit().await; + caches.clear_radix(); + res + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::storage::InMemoryStorage; + + fn wrapped(capacity: usize) -> Arc> { + CachedStorage::wrap( + Box::new(InMemoryStorage::default()) as Box, + capacity, + ) + } + + #[tokio::test] + async fn cache_serves_repeated_get_node_without_inner() { + let s = wrapped(64); + let id = { + let mut st = s.write().await; + st.new_node(b"hello".to_vec(), 42).await.unwrap() + }; + // First read misses (populates), second should hit cache — both correct. + { + let st = s.read().await; + assert_eq!(st.get_node(id).await.unwrap(), (b"hello".to_vec(), 42)); + } + { + let st = s.read().await; + assert_eq!(st.get_node(id).await.unwrap(), (b"hello".to_vec(), 42)); + } + } + + #[tokio::test] + async fn update_invalidates_node_cache() { + let s = wrapped(64); + let id = { + let mut st = s.write().await; + st.new_node(b"init".to_vec(), 1).await.unwrap() + }; + { + let st = s.read().await; + assert_eq!(st.get_node(id).await.unwrap().0, b"init".to_vec()); + } + { + let mut st = s.write().await; + st.update_node(id, Some(b"updated".to_vec()), Some(99)) + .await + .unwrap(); + } + // After update, cache must reflect new value (not stale). + let st = s.read().await; + assert_eq!(st.get_node(id).await.unwrap(), (b"updated".to_vec(), 99)); + } + + #[tokio::test] + async fn tx_commit_invalidates_radix_cache() { + let s = wrapped(64); + let parent = { + let mut st = s.write().await; + st.new_node(b"p".to_vec(), 0).await.unwrap() + }; + let child = { + let mut st = s.write().await; + st.new_node(b"c".to_vec(), 1).await.unwrap() + }; + // Pre-populate children cache. + { + let st = s.read().await; + assert!(st.get_children(parent).await.unwrap().is_empty()); + } + // Add child via tx, then commit → children cache must be invalidated. + { + let st = s.write().await; + let mut tx = st.new_tx(); + tx.add_child(parent, child).await.unwrap(); + tx.commit().await.unwrap(); + } + let st = s.read().await; + let children = st.get_children(parent).await.unwrap(); + assert!(children.contains(&child), "children after tx: {children:?}"); + } + + #[tokio::test] + async fn cache_matches_inner_semantics() { + let s = wrapped(128); + { + let mut st = s.write().await; + st.set_meta(7, b"meta-7".as_slice()).await.unwrap(); + st.set_key_len(7, 5).await.unwrap(); + st.set_chain(9, &[1, 2, 3]).await.unwrap(); + st.set_edge_data(3, b"edge-3").await.unwrap(); + st.set_node_meta(4, b"nm-4").await.unwrap(); + } + let st = s.read().await; + assert_eq!( + st.get_meta(7).await.unwrap().as_deref(), + Some(b"meta-7".as_slice()) + ); + assert_eq!(st.get_key_len(7).await.unwrap(), Some(5)); + assert_eq!(st.get_chain(9).await.unwrap(), Some(vec![1, 2, 3])); + assert_eq!( + st.get_edge_data(3).await.unwrap().as_deref(), + Some(b"edge-3".as_slice()) + ); + assert_eq!( + st.get_node_meta(4).await.unwrap().as_deref(), + Some(b"nm-4".as_slice()) + ); + // Second read hits cache, same result. + assert_eq!( + st.get_meta(7).await.unwrap().as_deref(), + Some(b"meta-7".as_slice()) + ); + } +} diff --git a/crates/codegraph-graph/src/storage/lmdb.rs b/crates/codegraph-graph/src/storage/lmdb.rs index cfc74f231..7b020a0ba 100644 --- a/crates/codegraph-graph/src/storage/lmdb.rs +++ b/crates/codegraph-graph/src/storage/lmdb.rs @@ -24,8 +24,8 @@ use lmdb::EnvironmentFlags; use lmdb::{Cursor, Database, DatabaseFlags, Environment, Transaction, WriteFlags}; use super::{ - EMPTY, Result, Storage, StorageError, Tx, TxOp, decode_chain, decode_vector, encode_chain, - encode_vector, + EMPTY, IndexCounts, Result, Storage, StorageError, Tx, TxOp, decode_chain, decode_vector, + encode_chain, encode_vector, }; /// Map lỗi LMDB → `StorageError`. @@ -50,6 +50,28 @@ fn de_u64(b: &[u8]) -> u64 { u64::from_le_bytes(b.try_into().expect("8-byte value")) } +/// Pack `IndexCounts` (5 × u64 LE) thành 40-byte value — lưu gọn trong 1 key. +fn pack_counts(c: &IndexCounts) -> [u8; 40] { + let mut b = [0u8; 40]; + b[0..8].copy_from_slice(&c.symbols.to_le_bytes()); + b[8..16].copy_from_slice(&c.chains.to_le_bytes()); + b[16..24].copy_from_slice(&c.edges.to_le_bytes()); + b[24..32].copy_from_slice(&c.files.to_le_bytes()); + b[32..40].copy_from_slice(&c.next_id.to_le_bytes()); + b +} + +fn unpack_counts(b: &[u8]) -> IndexCounts { + let at = |i: usize| u64::from_le_bytes(b[i..i + 8].try_into().expect("8-byte value")); + IndexCounts { + symbols: at(0), + chains: at(8), + edges: at(16), + files: at(24), + next_id: at(32), + } +} + // ── key chuỗi dài ── // // LMDB giới hạn key ≈ 511 byte (MDB_BAD_VALSIZE nếu vượt). Hai DBI dùng key là @@ -154,6 +176,7 @@ const D_CALL_NAMES: &str = "sg_call_names"; const D_FILES: &str = "sg_files"; const D_VERSION: &str = "sg_meta"; const D_EMBEDDINGS: &str = "sg_embeddings"; +const D_STATS: &str = "sg_stats"; /// Key duy nhất cho các "row đơn" (counter / next_id / version) — mỗi DBI chỉ có 1 row. const KEY_ONE: [u8; 8] = [0u8; 8]; @@ -177,6 +200,9 @@ fn open_env_read_only(path: &str) -> lmdb::Result { let mut b = Environment::new(); b.set_flags(EnvironmentFlags::READ_ONLY); b.set_max_dbs(32); + // Phải set map_size khớp với env read-write (1 GiB) — mở read-only không set + // map_size có thể trả EACCES/PERMISSION_DENIED trên một số platform. + b.set_map_size(1 << 30); b.open(Path::new(path)) } @@ -250,6 +276,7 @@ pub struct LmdbStorage { files: Database, version: Database, embeddings: Database, + stats: Database, } impl LmdbStorage { @@ -318,6 +345,9 @@ impl LmdbStorage { let embeddings = env .create_db(Some(D_EMBEDDINGS), DatabaseFlags::empty()) .map_err(e)?; + let stats = env + .create_db(Some(D_STATS), DatabaseFlags::empty()) + .map_err(e)?; Ok(Self { env, nodes, @@ -339,6 +369,7 @@ impl LmdbStorage { files, version, embeddings, + stats, }) } @@ -363,6 +394,10 @@ impl LmdbStorage { tx.put(self.version, &KEY_ONE, &ku64(0), WriteFlags::empty()) .map_err(e)?; } + if matches!(tx.get(self.stats, &KEY_ONE), Err(lmdb::Error::NotFound)) { + tx.put(self.stats, &KEY_ONE, &[0u8; 40], WriteFlags::empty()) + .map_err(e)?; + } tx.commit().map_err(e)?; Ok(()) } @@ -759,6 +794,23 @@ impl Storage for LmdbStorage { Ok(()) } + async fn set_stats(&mut self, s: IndexCounts) -> Result<()> { + let mut tx = self.env.begin_rw_txn().map_err(e)?; + tx.put(self.stats, &KEY_ONE, &pack_counts(&s), WriteFlags::empty()) + .map_err(e)?; + tx.commit().map_err(e)?; + Ok(()) + } + + async fn stats(&self) -> Result { + let tx = self.env.begin_ro_txn().map_err(e)?; + match tx.get(self.stats, &KEY_ONE) { + Ok(b) => Ok(unpack_counts(b)), + Err(lmdb::Error::NotFound) => Ok(IndexCounts::default()), + Err(err) => Err(StorageError::Internal(err.to_string())), + } + } + async fn clear_entities(&mut self) -> Result<()> { let mut tx = self.env.begin_rw_txn().map_err(e)?; for db in [ @@ -1218,4 +1270,26 @@ mod tests { assert_eq!(files.len(), 1); assert_eq!(files[0].path, long_path); } + + #[tokio::test] + async fn test_stats_roundtrip() { + let (_d, path) = tmp_path(); + let mut s = LmdbStorage::open(&path).await.unwrap(); + // Chưa ghi → trả zeros (caller fallback rebuild). + assert_eq!(s.stats().await.unwrap(), IndexCounts::default()); + let counts = IndexCounts { + symbols: 12, + chains: 3, + edges: 5, + files: 2, + next_id: 100, + }; + s.set_stats(counts).await.unwrap(); + assert_eq!(s.stats().await.unwrap(), counts); + drop(s); + // Mở lại (stats_cached mở storage từ route — LMDB cho phép nhiều RW handle) + // vẫn đọc được counts đã persist. + let ro = LmdbStorage::open(&path).await.unwrap(); + assert_eq!(ro.stats().await.unwrap(), counts); + } } diff --git a/crates/codegraph-graph/src/storage/mysql.rs b/crates/codegraph-graph/src/storage/mysql.rs index 296da3793..f8fa93187 100644 --- a/crates/codegraph-graph/src/storage/mysql.rs +++ b/crates/codegraph-graph/src/storage/mysql.rs @@ -1,7 +1,8 @@ use std::collections::HashMap; use super::{ - Result, Storage, StorageError, Tx, decode_chain, decode_vector, encode_chain, encode_vector, + IndexCounts, Result, Storage, StorageError, Tx, decode_chain, decode_vector, encode_chain, + encode_vector, }; use async_trait::async_trait; use codegraph_core::{Annotation, FileInfo, ScopeLevel, Symbol, SymbolKind}; @@ -76,6 +77,21 @@ impl MySqlStorage { .execute(&self.pool) .await .map_err(db_err)?; + // Stats tổng hợp (codegraph_status đọc O(1) không rebuild) — idempotent. + sqlx::query( + "CREATE TABLE IF NOT EXISTS sg_stats ( + repo_id BIGINT NOT NULL, + symbols BIGINT NOT NULL, + chains BIGINT NOT NULL, + edges BIGINT NOT NULL, + files BIGINT NOT NULL, + next_id BIGINT NOT NULL, + PRIMARY KEY (repo_id) + )", + ) + .execute(&self.pool) + .await + .map_err(db_err)?; Ok(()) } @@ -708,6 +724,46 @@ impl Storage for MySqlStorage { Ok(()) } + async fn set_stats(&mut self, s: IndexCounts) -> Result<()> { + sqlx::query( + "INSERT INTO sg_stats (repo_id, symbols, chains, edges, files, next_id) \ + VALUES (?, ?, ?, ?, ?, ?) \ + ON DUPLICATE KEY UPDATE \ + symbols = VALUES(symbols), chains = VALUES(chains), \ + edges = VALUES(edges), files = VALUES(files), next_id = VALUES(next_id)", + ) + .bind(self.repo_id as i64) + .bind(s.symbols as i64) + .bind(s.chains as i64) + .bind(s.edges as i64) + .bind(s.files as i64) + .bind(s.next_id as i64) + .execute(&self.pool) + .await + .map_err(db_err)?; + Ok(()) + } + + async fn stats(&self) -> Result { + let row: Option<(i64, i64, i64, i64, i64)> = sqlx::query_as( + "SELECT symbols, chains, edges, files, next_id FROM sg_stats WHERE repo_id = ?", + ) + .bind(self.repo_id as i64) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + match row { + Some((symbols, chains, edges, files, next_id)) => Ok(IndexCounts { + symbols: symbols as u64, + chains: chains as u64, + edges: edges as u64, + files: files as u64, + next_id: next_id as u64, + }), + None => Ok(IndexCounts::default()), + } + } + async fn clear_entities(&mut self) -> Result<()> { let rid = self.repo_id as i64; let mut tx = self.pool.begin().await.map_err(db_err)?; diff --git a/crates/codegraph-graph/src/storage/postgres.rs b/crates/codegraph-graph/src/storage/postgres.rs index 234f4f3ab..665cb15be 100644 --- a/crates/codegraph-graph/src/storage/postgres.rs +++ b/crates/codegraph-graph/src/storage/postgres.rs @@ -1,7 +1,8 @@ use std::collections::HashMap; use super::{ - Result, Storage, StorageError, Tx, decode_chain, decode_vector, encode_chain, encode_vector, + IndexCounts, Result, Storage, StorageError, Tx, decode_chain, decode_vector, encode_chain, + encode_vector, }; use async_trait::async_trait; use codegraph_core::{Annotation, FileInfo, ScopeLevel, Symbol, SymbolKind}; @@ -88,6 +89,21 @@ impl PostgresStorage { .execute(&self.pool) .await .map_err(db_err)?; + // Stats tổng hợp (codegraph_status đọc O(1) không rebuild) — idempotent. + sqlx::query( + "CREATE TABLE IF NOT EXISTS sg_stats ( + repo_id BIGINT NOT NULL, + symbols BIGINT NOT NULL, + chains BIGINT NOT NULL, + edges BIGINT NOT NULL, + files BIGINT NOT NULL, + next_id BIGINT NOT NULL, + PRIMARY KEY (repo_id) + )", + ) + .execute(&self.pool) + .await + .map_err(db_err)?; Ok(()) } @@ -720,6 +736,46 @@ impl Storage for PostgresStorage { Ok(()) } + async fn set_stats(&mut self, s: IndexCounts) -> Result<()> { + sqlx::query( + "INSERT INTO sg_stats (repo_id, symbols, chains, edges, files, next_id) \ + VALUES ($1, $2, $3, $4, $5, $6) \ + ON CONFLICT (repo_id) DO UPDATE SET \ + symbols = EXCLUDED.symbols, chains = EXCLUDED.chains, \ + edges = EXCLUDED.edges, files = EXCLUDED.files, next_id = EXCLUDED.next_id", + ) + .bind(self.repo_id as i64) + .bind(s.symbols as i64) + .bind(s.chains as i64) + .bind(s.edges as i64) + .bind(s.files as i64) + .bind(s.next_id as i64) + .execute(&self.pool) + .await + .map_err(db_err)?; + Ok(()) + } + + async fn stats(&self) -> Result { + let row: Option<(i64, i64, i64, i64, i64)> = sqlx::query_as( + "SELECT symbols, chains, edges, files, next_id FROM sg_stats WHERE repo_id = $1", + ) + .bind(self.repo_id as i64) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + match row { + Some((symbols, chains, edges, files, next_id)) => Ok(IndexCounts { + symbols: symbols as u64, + chains: chains as u64, + edges: edges as u64, + files: files as u64, + next_id: next_id as u64, + }), + None => Ok(IndexCounts::default()), + } + } + async fn clear_entities(&mut self) -> Result<()> { let rid = self.repo_id as i64; let mut tx = self.pool.begin().await.map_err(db_err)?; diff --git a/crates/codegraph-graph/src/storage/sqlite.rs b/crates/codegraph-graph/src/storage/sqlite.rs index 1cba8a77b..9e4d70b83 100644 --- a/crates/codegraph-graph/src/storage/sqlite.rs +++ b/crates/codegraph-graph/src/storage/sqlite.rs @@ -42,7 +42,9 @@ use codegraph_core::{FileInfo, Symbol}; use sqlx::Row; use sqlx::sqlite::{SqliteConnectOptions, SqliteJournalMode, SqlitePool, SqlitePoolOptions}; -use super::{EMPTY, Result, Storage, StorageError, Tx, TxOp, decode_vector, encode_vector}; +use super::{ + EMPTY, IndexCounts, Result, Storage, StorageError, Tx, TxOp, decode_vector, encode_vector, +}; use crate::embeddings::resolve_vss_extensions; fn db_err(e: sqlx::Error) -> StorageError { @@ -218,6 +220,15 @@ impl SqliteStorage { id INTEGER PRIMARY KEY CHECK (id = 1), version INTEGER NOT NULL )", + // ── Stats (counts tổng hợp — codegraph_status đọc O(1)) ── + "CREATE TABLE IF NOT EXISTS sg_stats ( + id INTEGER PRIMARY KEY CHECK (id = 1), + symbols INTEGER NOT NULL, + chains INTEGER NOT NULL, + edges INTEGER NOT NULL, + files INTEGER NOT NULL, + next_id INTEGER NOT NULL + )", // ── Embeddings (vector per symbol id) ── "CREATE TABLE IF NOT EXISTS sg_embeddings ( symbol_id INTEGER PRIMARY KEY, @@ -229,6 +240,7 @@ impl SqliteStorage { // next_id bắt đầu từ SYMBOL_BASE (marker reserved 1..=99). "INSERT OR IGNORE INTO sg_next_id (id, next) VALUES (1, 100)", "INSERT OR IGNORE INTO sg_meta (id, version) VALUES (1, 0)", + "INSERT OR IGNORE INTO sg_stats (id, symbols, chains, edges, files, next_id) VALUES (1, 0, 0, 0, 0, 0)", ] { sqlx::query(stmt) .execute(&mut *conn) @@ -745,6 +757,47 @@ impl Storage for SqliteStorage { Ok(()) } + async fn set_stats(&mut self, s: IndexCounts) -> Result<()> { + let mut conn = self.pool.acquire().await.map_err(db_err)?; + sqlx::query( + "INSERT INTO sg_stats (id, symbols, chains, edges, files, next_id) \ + VALUES (1, ?1, ?2, ?3, ?4, ?5) \ + ON CONFLICT(id) DO UPDATE SET \ + symbols = excluded.symbols, chains = excluded.chains, \ + edges = excluded.edges, files = excluded.files, next_id = excluded.next_id", + ) + .bind(s.symbols as i64) + .bind(s.chains as i64) + .bind(s.edges as i64) + .bind(s.files as i64) + .bind(s.next_id as i64) + .execute(&mut *conn) + .await + .map_err(db_err)?; + Ok(()) + } + + async fn stats(&self) -> Result { + let mut conn = self.pool.acquire().await.map_err(db_err)?; + let row: Option<(i64, i64, i64, i64, i64)> = sqlx::query_as( + "SELECT symbols, chains, edges, files, next_id FROM sg_stats WHERE id = 1", + ) + .fetch_optional(&mut *conn) + .await + .map_err(db_err)?; + match row { + Some((symbols, chains, edges, files, next_id)) => Ok(IndexCounts { + symbols: symbols as u64, + chains: chains as u64, + edges: edges as u64, + files: files as u64, + next_id: next_id as u64, + }), + // Bảng thiếu (index cũ) → trả 0 để caller fallback rebuild. + None => Ok(IndexCounts::default()), + } + } + async fn clear_entities(&mut self) -> Result<()> { let mut conn = self.pool.acquire().await.map_err(db_err)?; for stmt in [ @@ -1386,4 +1439,35 @@ mod tests { let n = s.new_node(b"new".to_vec(), 1).await.unwrap(); assert!(n > parent); } + + #[tokio::test] + async fn test_stats_roundtrip() { + let (_d, path) = tmp_path(); + let mut s = SqliteStorage::open(&path).await.unwrap(); + s.init().await.unwrap(); + // Chưa ghi → trả zeros (caller fallback rebuild). + assert_eq!(s.stats().await.unwrap(), IndexCounts::default()); + let counts = IndexCounts { + symbols: 12, + chains: 3, + edges: 5, + files: 2, + next_id: 100, + }; + s.set_stats(counts).await.unwrap(); + let got = s.stats().await.unwrap(); + assert_eq!(got.symbols, 12); + assert_eq!(got.chains, 3); + assert_eq!(got.edges, 5); + assert_eq!(got.files, 2); + assert_eq!(got.next_id, 100); + // Ghi lại đè → UPSERT cập nhật (không duplicate row). + s.set_stats(IndexCounts { + symbols: 99, + ..IndexCounts::default() + }) + .await + .unwrap(); + assert_eq!(s.stats().await.unwrap().symbols, 99); + } } diff --git a/crates/codegraph-installer/src/instructions-template.md b/crates/codegraph-installer/src/instructions-template.md index cc5111c1c..62d117f2b 100644 --- a/crates/codegraph-installer/src/instructions-template.md +++ b/crates/codegraph-installer/src/instructions-template.md @@ -1,7 +1,7 @@ # CodeGraph This project has a CodeGraph MCP server configured. CodeGraph is a tree-sitter -knowledge graph of every symbol, edge, and file in the workspace. Reads are +semantic graph of every symbol and call chain in the workspace. Reads are sub-millisecond and return structural information grep cannot. ## When to prefer codegraph @@ -12,11 +12,12 @@ for literal text queries. | Question | Tool | |---|---| -| "Where is X defined?" | `codegraph_search` | +| "Where is X defined?" | `codegraph_search_symbol` | +| "Show me X by id / exact name" | `codegraph_symbol` | | "What calls Y?" | `codegraph_callers` | | "What does Y call?" | `codegraph_callees` | | "What would break if I changed Z?" | `codegraph_impact` | -| "Show me Y's signature / source" | `codegraph_node` | +| "Show me Y's call chain" | `codegraph_flow` | | "Give me focused context for a task" | `codegraph_context` | | "What files exist under path/" | `codegraph_files` | | "Is the index healthy?" | `codegraph_status` | @@ -26,9 +27,12 @@ for literal text queries. - **Trust codegraph results.** They come from a full AST parse. Do NOT re-verify with grep. - **Don't grep first** when looking up a symbol by name. -- **`codegraph_context` is one call** — don't chain search + node yourself. +- **`codegraph_context` is one call** — don't chain search + symbol yourself. +- **Duplicate names** return `ambiguous: true` — retry with the numeric `id`. -## If `.codegraph/` doesn't exist +## If no index exists yet -The MCP server returns "not initialized." Run `codegraph init -i` to build -the index. +Query tools refuse until the session is bound. Call +`codegraph_init {"path": ...}` (non-blocking — does NOT index by default), +then `codegraph_index {}` to build the index. On the CLI, `codegraph init` +creates `.codegraph/` and indexes in one step. diff --git a/crates/codegraph-installer/src/targets/antigravity.rs b/crates/codegraph-installer/src/targets/antigravity.rs index 1eedc801f..faa0ea8f6 100644 --- a/crates/codegraph-installer/src/targets/antigravity.rs +++ b/crates/codegraph-installer/src/targets/antigravity.rs @@ -162,33 +162,48 @@ paths: # CodeGraph System Instructions -This project is backed by a custom **CodeGraph MCP server**. CodeGraph maintains a local Tree-sitter knowledge graph encompassing every symbol, edge, boundary, and file within this workspace. Reads operate at sub-millisecond speeds and deliver accurate structural insights that traditional text-based tools (like grep) cannot match. +This project is backed by a **CodeGraph MCP server** — a local tree-sitter +semantic graph of every symbol and call chain in the workspace. Reads are +sub-millisecond and return structural information grep cannot match. --- ## 🚨 CRITICAL CONSTRAINTS (Read First) -- **NEVER use generic text-search, grep, or file-reading tools** if a symbol, reference, or definition can be located using CodeGraph. -- **DO NOT double-check or re-verify** CodeGraph results with native file reads. Treat the knowledge graph as the absolute, single source of truth for codebase architecture. -- **Handle uninitialized states immediately:** If any CodeGraph tool returns a `"not initialized"` or missing index error, **STOP execution immediately** and instruct the user to run `codegraph init -i` in their terminal. Do not attempt to parse or scan the codebase manually to compensate. -- **Minimize token overhead:** Prefer targeted structural queries over dumping entire file contents into the context window. +- **NEVER use generic text-search, grep, or file-reading tools** when a + symbol, reference, or definition can be located with CodeGraph. +- **Do NOT re-verify** CodeGraph results with file reads — the graph is the + single source of truth for codebase structure. +- **Handle unbound sessions:** query tools refuse until the session is bound. + Call `codegraph_init {"path": ...}` (non-blocking, does NOT index), then + `codegraph_index {}` to build/refresh the index. +- **Minimize token overhead:** prefer targeted structural queries and + `id`-based lookups over dumping file contents into context. --- ## 🛠️ Tool Selection Guide -Always prefer `codegraph` tools for **structural** questions — tracing call hierarchies, mapping dependencies, determining definitions, and verifying signatures. Use standard filesystem tools *only* for literal text queries or applying actual code edits. +Prefer codegraph for **structural** questions. Use filesystem tools only for +literal text queries or applying edits. | Intent / Question | Recommended MCP Tool | | :--- | :--- | -| *"Where is symbol X defined?"* | `codegraph_search` | +| *"Where is symbol X defined?"* | `codegraph_search_symbol` (contains/prefix/suffix/exact) | +| *"Show me this symbol by id / exact name"* | `codegraph_symbol` | | *"What callers invoke function Y?"* | `codegraph_callers` | | *"What methods or functions does Y call?"* | `codegraph_callees` | | *"What components or files will break if I modify Z?"* | `codegraph_impact` | -| *"Show me Y's exact signature and internal block"* | `codegraph_node` | +| *"Show me Y's call chain"* | `codegraph_flow` | +| *"Find flows containing a pattern (loop + call)"* | `codegraph_search_flow` | | *"Give me focused, aggregated context for this task"* | `codegraph_context` | +| *"Who calls the library function foo?"* | `codegraph_references` | +| *"What fields/methods does class C have?"* | `codegraph_class` | +| *"Which symbols are annotated @X?"* | `codegraph_search_by_annotation` | | *"What files exist under a specific path/ directory?"* | `codegraph_files` | | *"Is the local knowledge graph healthy and active?"* | `codegraph_status` | +| *"What does this MR change in the graph?"* | `codegraph_diff` | +| *"Simulate a flow's behavior with mocks"* | `codegraph_sandbox` / `codegraph_diff_simulate` | --- @@ -202,4 +217,8 @@ Before updating an API route, a UI component, or a system utility, query `codegr ### 3. Strict AST Reliance Because CodeGraph parses the Abstract Syntax Tree (AST), its structural insights are guaranteed. If a symbol look-up yields no results, assume the symbol does not exist in the current active workspace index. + +### 4. Duplicate names +The tool returns `ambiguous: true` with the full match list — retry with the +numeric `id` alone. "#; diff --git a/crates/codegraph-mcp/src/lib.rs b/crates/codegraph-mcp/src/lib.rs index 8ce305d01..f2148506f 100644 --- a/crates/codegraph-mcp/src/lib.rs +++ b/crates/codegraph-mcp/src/lib.rs @@ -35,7 +35,8 @@ use rmcp::{ErrorData as McpError, RoleServer}; use serde_json::{json, Value}; /// Hướng dẫn sử dụng tools — client render trong instructions sau `initialize`. -pub const SERVER_INSTRUCTIONS: &str = include_str!("server-instructions.md"); +/// Nguồn chân lý nằm ở `docs/codegraph.md` (gom chung tài liệu trong docs/). +pub const SERVER_INSTRUCTIONS: &str = include_str!("../../../docs/codegraph.md"); pub const SERVER_NAME: &str = "codegraph"; pub const SERVER_VERSION: &str = env!("CARGO_PKG_VERSION"); diff --git a/crates/codegraph-mcp/src/tools.rs b/crates/codegraph-mcp/src/tools.rs index dbea9c5a8..e5aa3bd9b 100644 --- a/crates/codegraph-mcp/src/tools.rs +++ b/crates/codegraph-mcp/src/tools.rs @@ -494,7 +494,7 @@ pub async fn dispatch_with_api( emit(root.as_str(), &files) } "codegraph_status" => { - let stats = api.stats().await; + let stats = api.stats_cached().await; emit(root.as_str(), &stats) } "codegraph_search_symbol" => { @@ -981,7 +981,7 @@ fn format_from_args(args: &Value, session: OutputStyle) -> OutputStyle { } /// Symbol JSON theo `detail` + `style`. `Minimize` (mặc định) → mảng vị trí cố -/// định (order được document trong server-instructions.md; file đã relativize +/// định (order được document trong docs/codegraph.md; file đã relativize /// theo root — relativize_paths chỉ chạm object key, không chạm phần tử mảng); /// `Medium` → object giữ key (field default bị lược sau trong `omit_defaults`). fn symbol_json(root: &str, s: &Symbol, detail: DetailLevel, style: OutputStyle) -> Value { diff --git a/crates/codesmell/Cargo.toml b/crates/codesmell/Cargo.toml new file mode 100644 index 000000000..9eb7a70f1 --- /dev/null +++ b/crates/codesmell/Cargo.toml @@ -0,0 +1,25 @@ +[package] +name = "codesmell" +version.workspace = true +edition.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +codegraph-core = { path = "../codegraph-core" } +codegraph-extract = { path = "../codegraph-extract" } +codegraph-graph = { path = "../codegraph-graph" } +serde = { workspace = true } +serde_json = { workspace = true } +toml = "0.8" +camino = { workspace = true } +globset = { workspace = true } +anyhow = { workspace = true } +dirs = "5" +clap = { workspace = true } +tokio = { workspace = true } +rhai = { version = "1", features = ["sync"] } +regex = "1" + +[dev-dependencies] +tempfile = "3" diff --git a/crates/codesmell/packs/security/policy.fragment.toml b/crates/codesmell/packs/security/policy.fragment.toml new file mode 100644 index 000000000..3eaab9486 --- /dev/null +++ b/crates/codesmell/packs/security/policy.fragment.toml @@ -0,0 +1,26 @@ +# CodeSmell security pack — installed by `codesmell pack add security`. +# Edit freely; delete `.codesmell/packs/security.policy.toml` and the scripts in +# `.codesmell/rules/` to remove the pack. Merge order: entries are appended to +# your policy; `severity` entries here win over the same id elsewhere. + +[severity] +"security.deny_call" = "required" +"security.deny_symbol" = "blocking" + +[[rhai.rule]] +use = "security.deny_call" +params = { deny = ["eval"], deny_re = ['(?i)^(os\.)?system$', '(?i)^child_process$'], message = "dynamic code execution", fix_hint = "parse the input explicitly instead of executing strings" } + +[[rhai.rule]] +use = "security.deny_symbol" +params = { kind = "constant", name_re = ['(?i)^(PASSWORD|API_KEY|SECRET|AWS_ACCESS_KEY|TOKEN|PRIVATE_KEY)$'], message = "hard-coded secret-like name", fix_hint = "load from env vars / a secret manager" } + +[[rhai.rule]] +use = "security.dangerous_exec" +params = { skip_files_containing = ["tests/", "spec/", "_test", "test_"] } + +[[rhai.rule]] +use = "security.crypto_weak_hash" + +[[rhai.rule]] +use = "security.dangerous_deserialize" diff --git a/crates/codesmell/packs/security/rules/security.crypto_weak_hash.rhai b/crates/codesmell/packs/security/rules/security.crypto_weak_hash.rhai new file mode 100644 index 000000000..323fe03a3 --- /dev/null +++ b/crates/codesmell/packs/security/rules/security.crypto_weak_hash.rhai @@ -0,0 +1,30 @@ +// security.crypto_weak_hash — weak hashes and non-CSPRNGs for security values. +const ADVICE = "Use SHA-256+ for hashes and a CSPRNG for tokens — never md5/sha1/Math.random."; + +fn check_calls(sym, callees, callers) { + let weak = callees.filter(|c| { + let l = c.name.to_lower(); + l.contains("md5") || l.contains("sha1") || l == "sha-1" + || l.contains("digest_md5") || l.contains("md5crypt") + }); + let risky_rand = (callees.filter(|c| { + let l = c.name.to_lower(); + l.contains("random") || l.contains("rand") + }).len() > 0) + && (sym.name.to_lower().contains("token") || sym.name.to_lower().contains("secret") + || sym.name.to_lower().contains("nonce") || sym.name.to_lower().contains("password") + || sym.name.to_lower().contains("session") || sym.name.to_lower().contains("key")); + if weak.len() > 0 { + return #{ + message: "`" + sym.name + "` uses weak hash `" + weak[0].name + "`", + hint: "use SHA-256 or stronger" + }; + } + if risky_rand { + return #{ + message: "`" + sym.name + "` uses a non-cryptographic RNG for security values", + hint: "use a CSPRNG (e.g. crypto/rand, secrets module)" + }; + } + false +} diff --git a/crates/codesmell/packs/security/rules/security.dangerous_deserialize.rhai b/crates/codesmell/packs/security/rules/security.dangerous_deserialize.rhai new file mode 100644 index 000000000..e33ecaf72 --- /dev/null +++ b/crates/codesmell/packs/security/rules/security.dangerous_deserialize.rhai @@ -0,0 +1,21 @@ +// security.dangerous_deserialize — unsafe native deserialization. +const ADVICE = "Never deserialize untrusted data with native formats (pickle/yaml.load/Marshal/readObject)."; + +fn check_calls(sym, callees, callers) { + let sinks = ["pickle.loads", "pickle.load", "yaml.load", "marshal.load", "unserialize", "read_object", "objectinputstream"]; + let hit = callees.filter(|c| { + let l = c.name.to_lower(); + let found = false; + for k in sinks { + if l.contains(k) { found = true; break; } + } + found + }); + if hit.len() > 0 { + return #{ + message: "`" + sym.name + "` deserializes with `" + hit[0].name + "`", + hint: "use a data-only format (JSON) with schema validation" + }; + } + false +} diff --git a/crates/codesmell/packs/security/rules/security.dangerous_exec.rhai b/crates/codesmell/packs/security/rules/security.dangerous_exec.rhai new file mode 100644 index 000000000..e23c74104 --- /dev/null +++ b/crates/codesmell/packs/security/rules/security.dangerous_exec.rhai @@ -0,0 +1,26 @@ +// security.dangerous_exec — dynamic execution sinks (exec/system/popen/spawn). +// Skips files matching `params.skip_files_containing` (e.g. tests that shell out on purpose). +const ADVICE = "Never shell out via exec/system/popen with untrusted input — use explicit APIs."; + +fn check_calls(sym, callees, callers) { + let skip = params.skip_files_containing ?? []; + for s in skip { + if sym.file.contains(s) { return false; } + } + let sinks = ["exec", "system", "popen", "spawn", "shell_exec", "run_command"]; + let hit = callees.filter(|c| { + let l = c.name.to_lower(); + let found = false; + for k in sinks { + if l.contains(k) { found = true; break; } + } + found + }); + if hit.len() > 0 { + return #{ + message: "`" + sym.name + "` calls `" + hit[0].name + "` (dynamic execution sink)", + hint: "replace with an explicit API call or a proper parser" + }; + } + false +} diff --git a/crates/codesmell/rules_builtin/architecture.boundary.rhai b/crates/codesmell/rules_builtin/architecture.boundary.rhai new file mode 100644 index 000000000..87776bb98 --- /dev/null +++ b/crates/codesmell/rules_builtin/architecture.boundary.rhai @@ -0,0 +1,41 @@ +// architecture.boundary — forbid calls that cross denied layer edges. +// Enabled via `[[rhai.rule]] use = "architecture.boundary" params = { +// layers = [{ name = "controller", paths = ["src/controllers/**"] }, +// { name = "repository", paths = ["src/repositories/**"] }], +// deny = ["controller -> repository"], +// }`. +const ADVICE = "Respect the architecture layer boundaries."; + +fn describe(_params) { "Architecture layer boundaries are enforced." } + +fn layer_of(layers, file) { + for l in layers { + for p in (l.paths ?? []) { + if glob(p, file) { return l.name; } + } + } + "" +} + +fn check_calls(sym, callees, callers) { + let layers = params.layers ?? []; + if layers.len() == 0 { return false; } + let deny = params.deny ?? []; + if deny.len() == 0 { return false; } + let caller_layer = layer_of(layers, sym.file); + if caller_layer == "" { return false; } + for callee in callees { + let callee_layer = layer_of(layers, callee.file); + if callee_layer == "" { continue; } + let edge = caller_layer + " -> " + callee_layer; + for d in deny { + if d == edge || d == (caller_layer + " -> *") { + return #{ + message: "`" + sym.name + "` (" + caller_layer + ") calls `" + callee.name + "` (" + callee_layer + "): edge `" + edge + "` is denied", + hint: "route `" + sym.name + "` through an allowed layer" + }; + } + } + } + false +} diff --git a/crates/codesmell/rules_builtin/security.deny_call.rhai b/crates/codesmell/rules_builtin/security.deny_call.rhai new file mode 100644 index 000000000..d37da2c66 --- /dev/null +++ b/crates/codesmell/rules_builtin/security.deny_call.rhai @@ -0,0 +1,41 @@ +// security.deny_call — forbid functions from calling denied callees. +// Enabled via `[[rhai.rule]] use = "security.deny_call" params = { +// deny = ["eval", "exec", "system"], +// deny_re = ['(?i)^(os\.system|child_process\.exec)$'], +// symbols = ["process_*"], symbols_re = ['(?i)^handle_.*webhook$'], +// message = "dynamic code execution", +// fix_hint = "parse the input explicitly instead of executing it", +// }`. +// (Entry-level `paths` / `exclude` scope which callers are checked.) +const ADVICE = "Avoid calling the denied functions."; + +fn describe(_params) { "Calls to specific dangerous functions are denied." } + +fn match_any(globs, regexes, text) { + for g in (globs ?? []) { + if glob(g, text) { return true; } + } + for r in (regexes ?? []) { + if regex_match(r, text) { return true; } + } + false +} + +fn check_calls(sym, callees, callers) { + let deny = params.deny ?? []; + let deny_re = params.deny_re ?? []; + let symbolsf = params.symbols ?? []; + let symbols_re = params.symbols_re ?? []; + if symbolsf.len() > 0 || symbols_re.len() > 0 { + if !match_any(symbolsf, symbols_re, sym.name) { return false; } + } + for callee in callees { + if match_any(deny, deny_re, callee.name) { + return #{ + message: params.message ?? ("call to `" + callee.name + "` is denied"), + hint: params.fix_hint ?? ("remove or replace the call to `" + callee.name + "`") + }; + } + } + false +} diff --git a/crates/codesmell/rules_builtin/security.deny_symbol.rhai b/crates/codesmell/rules_builtin/security.deny_symbol.rhai new file mode 100644 index 000000000..f712bae38 --- /dev/null +++ b/crates/codesmell/rules_builtin/security.deny_symbol.rhai @@ -0,0 +1,32 @@ +// security.deny_symbol — forbid symbol declarations matching denied patterns. +// Enabled via `[[rhai.rule]] use = "security.deny_symbol" params = { +// kind = "constant", # or "*" for any kind +// name = ["*Manager"], +// name_re = ['(?i)^(PASSWORD|API_KEY|SECRET|TOKEN)$'], +// message = "hard-coded secret-like name", +// fix_hint = "load from env vars / a secret manager", +// }`. +const ADVICE = "Avoid declaring symbols that match the denied patterns."; + +fn describe(_params) { "Certain symbol names/kinds are disallowed." } + +fn check(sym) { + let kind = params.kind ?? "*"; + if kind != "*" && sym.kind != kind { return false; } + let hit = false; + for g in (params.name ?? []) { + if glob(g, sym.name) { hit = true; break; } + } + if !hit { + for r in (params.name_re ?? []) { + if regex_match(r, sym.name) { hit = true; break; } + } + } + if hit { + return #{ + message: params.message ?? ("symbol `" + sym.name + "` matches a denied pattern"), + hint: params.fix_hint ?? "rename the symbol" + }; + } + false +} diff --git a/crates/codesmell/rules_builtin/style.function.max_complexity.rhai b/crates/codesmell/rules_builtin/style.function.max_complexity.rhai new file mode 100644 index 000000000..4dc06c1d1 --- /dev/null +++ b/crates/codesmell/rules_builtin/style.function.max_complexity.rhai @@ -0,0 +1,22 @@ +// style.function.max_complexity — keep cyclomatic complexity bounded. +// Enabled via `[[rhai.rule]] use = "style.function.max_complexity" params = { max = 10 }`. +// Complexity = 1 + number of decision markers (if_true/if_false/loop/switch_case). +const ADVICE = "Keep functions under the team complexity budget."; + +fn describe(params) { + "Cyclomatic complexity should stay at or below " + (params.max ?? 10).to_string() + "." +} + +fn check_flow(sym, markers) { + let max = params.max ?? 10; + let decisions = markers.filter(|m| + m == "if_true" || m == "if_false" || m == "loop" || m == "switch_case").len(); + let complexity = decisions + 1; + if complexity > max { + return #{ + message: "function `" + sym.name + "` has cyclomatic complexity " + complexity.to_string() + " (max " + max.to_string() + ")", + hint: "split the function into smaller units" + }; + } + false +} diff --git a/crates/codesmell/rules_builtin/style.function.max_lines.rhai b/crates/codesmell/rules_builtin/style.function.max_lines.rhai new file mode 100644 index 000000000..d6e3514c4 --- /dev/null +++ b/crates/codesmell/rules_builtin/style.function.max_lines.rhai @@ -0,0 +1,20 @@ +// style.function.max_lines — keep functions short and focused. +// Enabled via `[[rhai.rule]] use = "style.function.max_lines" params = { max = 60 }`. +const ADVICE = "Keep functions short and focused."; + +fn describe(params) { + "Functions normally stay below " + (params.max ?? 60).to_string() + " lines." +} + +fn check(sym) { + if sym.kind != "function" && sym.kind != "method" { return false; } + let max = params.max ?? 60; + let loc = sym.end_line - sym.line + 1; + if loc > max { + return #{ + message: "function `" + sym.name + "` is " + loc.to_string() + " lines (max " + max.to_string() + ")", + hint: "split `" + sym.name + "` into smaller functions" + }; + } + false +} diff --git a/crates/codesmell/rules_builtin/style.function.max_nesting.rhai b/crates/codesmell/rules_builtin/style.function.max_nesting.rhai new file mode 100644 index 000000000..7954e1f6b --- /dev/null +++ b/crates/codesmell/rules_builtin/style.function.max_nesting.rhai @@ -0,0 +1,28 @@ +// style.function.max_nesting — avoid deeply nested control flow. +// Enabled via `[[rhai.rule]] use = "style.function.max_nesting" params = { max = 4 }`. +const ADVICE = "Avoid deeply nested control flow."; + +fn describe(params) { + "Avoid nesting deeper than " + (params.max ?? 4).to_string() + " levels." +} + +fn check_flow(sym, markers) { + let max = params.max ?? 4; + let depth = 0; + let mx = 0; + for m in markers { + if m == "loop" || m == "if_true" || m == "if_false" || m == "switch_case" { + depth += 1; + if depth > mx { mx = depth; } + } else if m == "branch_end" || m == "loop_back" || m == "switch_end" || m == "break" || m == "continue" { + depth -= 1; + } + } + if mx > max { + return #{ + message: "function `" + sym.name + "` nesting depth is " + mx.to_string() + " (max " + max.to_string() + ")", + hint: "flatten early returns and extract nested blocks" + }; + } + false +} diff --git a/crates/codesmell/rules_builtin/style.function.max_parameters.rhai b/crates/codesmell/rules_builtin/style.function.max_parameters.rhai new file mode 100644 index 000000000..78ec0fb23 --- /dev/null +++ b/crates/codesmell/rules_builtin/style.function.max_parameters.rhai @@ -0,0 +1,76 @@ +// style.function.max_parameters — limit the number of real parameters. +// Enabled via `[[rhai.rule]] use = "style.function.max_parameters" params = { max = 4 }`. +const ADVICE = "Functions normally take at most a few real parameters."; + +fn describe(params) { + "Functions normally take at most " + (params.max ?? 4).to_string() + " parameters." +} + +// Strip leading/trailing ASCII spaces (rhai's `trim` is unreliable here). +fn strip(s) { + let n = s.len(); + let start = 0; + while start < n && s.sub_string(start, 1) == " " { start += 1; } + let end = n; + while end > start && s.sub_string(end - 1, 1) == " " { end -= 1; } + s.sub_string(start, end - start) +} + +fn real_param(seg) { + let t = strip(seg); + if t == "" { return false; } + if t == "self" || t == "&self" || t == "&mut self" { return false; } + if t.starts_with("self:") { return false; } + true +} + +fn count_params(sig) { + let open = sig.index_of("("); + if open < 0 { return 0; } + let depth = 0; + let end = -1; + let i = open; + while i < sig.len() { + let c = sig.sub_string(i, 1); + if c == "(" { depth += 1; } + else if c == ")" { + depth -= 1; + if depth == 0 { end = i; break; } + } + i += 1; + } + if end < 0 { return 0; } + let inner = sig.sub_string(open + 1, end - open - 1); + if strip(inner) == "" { return 0; } + // split on top-level commas only (commas inside (..) or <..> belong to one param) + let depth = 0; + let seg = ""; + let count = 0; + let chars = inner.len(); + let j = 0; + while j < chars { + let c = inner.sub_string(j, 1); + if c == "(" || c == "<" { depth += 1; seg += c; } + else if c == ")" || c == ">" { depth -= 1; seg += c; } + else if c == "," && depth == 0 { + if real_param(seg) { count += 1; } + seg = ""; + } else { seg += c; } + j += 1; + } + if real_param(seg) { count += 1; } + count +} + +fn check(sym) { + if sym.kind != "function" && sym.kind != "method" { return false; } + let max = params.max ?? 4; + let n = count_params(sym.signature); + if n > max { + return #{ + message: "function `" + sym.name + "` takes " + n.to_string() + " parameters (max " + max.to_string() + ")", + hint: "group parameters into a struct or options type" + }; + } + false +} diff --git a/crates/codesmell/rules_builtin/style.naming.rhai b/crates/codesmell/rules_builtin/style.naming.rhai new file mode 100644 index 000000000..6f051b197 --- /dev/null +++ b/crates/codesmell/rules_builtin/style.naming.rhai @@ -0,0 +1,34 @@ +// style.naming — enforce naming patterns per symbol kind. +// Enabled via `[[rhai.rule]] use = "style.naming" params = { rules = [ +// { kind = "method", pattern = "*Async", signature_contains = "async" }, +// { kind = "class", pattern = "*Service" }, +// ] }`. +const ADVICE = "Follow the team naming conventions."; + +fn describe(_params) { "Naming conventions are enforced." } + +fn check(sym) { + let rules = params.rules ?? []; + for r in rules { + let k = r.kind ?? ""; + if k != "" && sym.kind != k { continue; } + let sigc = r.signature_contains ?? ""; + if sigc != "" && !sym.signature.contains(sigc) { continue; } + let paths = r.paths ?? []; + if paths.len() > 0 { + let hit = false; + for p in paths { + if glob(p, sym.file) { hit = true; break; } + } + if !hit { continue; } + } + let ok = glob(r.pattern, sym.name); + if !ok { + return #{ + message: "`" + sym.name + "` should match naming pattern `" + r.pattern + "`", + hint: "rename to follow the team naming convention" + }; + } + } + false +} diff --git a/crates/codesmell/rules_builtin/testing.missing_test.rhai b/crates/codesmell/rules_builtin/testing.missing_test.rhai new file mode 100644 index 000000000..fe4f1995f --- /dev/null +++ b/crates/codesmell/rules_builtin/testing.missing_test.rhai @@ -0,0 +1,56 @@ +// testing.missing_test — business logic needs a unit test. +// Enabled via `[[rhai.rule]] use = "testing.missing_test" params = { +// require = true, +// test_paths = ["tests/**", "**/*_test.rs"], +// layers = [{ name = "service", paths = ["src/services/**"] }], +// selectors = [{ min_lines = 20 }], +// }`. +const ADVICE = "New or changed business logic requires a unit test."; + +fn describe(params) { + if params.require ?? false { "Business logic requires a unit test." } else { "" } +} + +fn layer_of(layers, file) { + for l in layers { + for p in (l.paths ?? []) { + if glob(p, file) { return l.name; } + } + } + "" +} + +fn is_logic(sym, layers, selectors) { + if selectors.len() == 0 { return true; } + let layer = layer_of(layers, sym.file); + for sel in selectors { + let sels = sel.layers ?? []; + for l in sels { + if layer == l { return true; } + } + let min = sel.min_lines ?? 0; + if min > 0 && (sym.end_line - sym.line + 1) >= min { return true; } + } + false +} + +fn check_calls(sym, callees, callers) { + if !(params.require ?? false) { return false; } + let test_paths = params.test_paths ?? []; + if test_paths.len() == 0 { return false; } + if !is_logic(sym, params.layers ?? [], params.selectors ?? []) { return false; } + let tested = false; + for caller in callers { + for p in test_paths { + if glob(p, caller.file) { tested = true; break; } + } + if tested { break; } + } + if !tested { + return #{ + message: "business logic `" + sym.name + "` has no unit test", + hint: "add a unit test that covers this logic" + }; + } + false +} diff --git a/crates/codesmell/src/engine.rs b/crates/codesmell/src/engine.rs new file mode 100644 index 000000000..e68f5465e --- /dev/null +++ b/crates/codesmell/src/engine.rs @@ -0,0 +1,132 @@ +//! Evaluation engine: collect symbols by kind, run the rhai rule instances, +//! produce a sorted report. + +use codegraph_core::SymbolKind; +use codegraph_graph::{diff::ParsedDiff, GraphIndex}; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +use crate::policy::Policy; +use crate::rhai; + +/// A single policy violation surfaced to the developer / LLM. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Violation { + pub rule: String, + pub severity: crate::policy::Severity, + pub file: String, + pub line: u32, + pub symbol: String, + pub message: String, + /// Action that fixes the violation — guides the LLM instead of leaving it + /// to guess. + pub fix_hint: String, +} + +/// Result of a `check` run. +#[derive(Debug, Clone, Serialize, Default)] +pub struct CheckReport { + pub violations: Vec, + /// Count of violations keyed by human label (`note`/`warning`/`error`). + pub summary: HashMap, +} + +/// What part of the repository to evaluate. +#[derive(Debug)] +pub enum CheckScope { + /// Whole repository (default). + All, + /// Symbols whose file is under one of the given paths. + Paths(Vec), + /// Symbols in files touched by a unified diff, within changed line ranges. + Diff(ParsedDiff), +} + +/// Repo-relative path for a symbol's file (used for layer / naming globs). +pub fn rel_path(file: &str, root: &Path) -> String { + Path::new(file) + .strip_prefix(root) + .map(|p| p.to_string_lossy().into_owned()) + .unwrap_or_else(|_| file.to_string()) +} + +/// Collect symbol candidates of the given `kinds`, narrowed by `scope`. +pub fn collect_symbols( + index: &GraphIndex, + kinds: &[SymbolKind], + scope: &CheckScope, + root: &Path, +) -> Vec { + let mut all = Vec::new(); + for k in kinds { + let (syms, _) = index.list_symbols_by_kind(*k, 0, 0); + all.append(&mut syms.into_iter().collect()); + } + + match scope { + CheckScope::All => all, + CheckScope::Paths(paths) => { + all.retain(|s| { + let target = Path::new(root).join(&s.file); + paths + .iter() + .any(|p| Path::new(&target).starts_with(Path::new(root).join(p))) + }); + all + } + CheckScope::Diff(parsed) => { + let mut changed: HashMap> = HashMap::new(); + for fd in &parsed.files { + let rel = fd + .path + .trim_start_matches("a/") + .trim_start_matches("b/") + .to_string(); + let entry = changed.entry(PathBuf::from(rel)).or_default(); + for h in &fd.hunks { + let lo = h.new_start; + let hi = h.new_start.saturating_add(h.new_len).saturating_sub(1); + for l in lo..=hi { + entry.insert(l); + } + for l in &h.new_lines { + entry.insert(*l); + } + } + } + all.retain(|s| { + let rel = PathBuf::from(rel_path(&s.file, root)); + match changed.get(&rel) { + Some(lines) => (s.line..=s.end_line).any(|l| lines.contains(&l)), + None => false, + } + }); + all + } + } +} + +/// Run the policy over the repository and return a severity-sorted report. +pub async fn evaluate( + index: &GraphIndex, + scope: &CheckScope, + policy: &Policy, + root: &Path, +) -> anyhow::Result { + let mut violations = rhai::run(index, scope, policy, root).await?; + + // Most serious first. + violations.sort_by_key(|v| std::cmp::Reverse(v.severity)); + + let mut summary: HashMap = HashMap::new(); + for v in &violations { + *summary + .entry(v.severity.as_label().to_string()) + .or_insert(0) += 1; + } + Ok(CheckReport { + violations, + summary, + }) +} diff --git a/crates/codesmell/src/glob.rs b/crates/codesmell/src/glob.rs new file mode 100644 index 000000000..bbf2f921a --- /dev/null +++ b/crates/codesmell/src/glob.rs @@ -0,0 +1,34 @@ +//! Small glob helpers built on `globset`. Patterns use glob syntax +//! (`*`, `**`, `?`); path globs are matched against repo-relative paths. + +use globset::{Glob, GlobMatcher}; + +/// Pre-compiled set of glob patterns, matched as a logical OR. +pub struct GlobSet { + matchers: Vec<(String, GlobMatcher)>, +} + +impl GlobSet { + pub fn new(patterns: &[String]) -> Self { + let matchers = patterns + .iter() + .filter_map(|p| Glob::new(p).ok().map(|g| (p.clone(), g.compile_matcher()))) + .collect(); + GlobSet { matchers } + } + + pub fn is_empty(&self) -> bool { + self.matchers.is_empty() + } + + pub fn matches(&self, path: &str) -> bool { + self.matchers.iter().any(|(_, m)| m.is_match(path)) + } +} + +/// One-shot glob match (compiles the pattern each call). +pub fn glob_matches(pattern: &str, path: &str) -> bool { + Glob::new(pattern) + .map(|g| g.compile_matcher().is_match(path)) + .unwrap_or(false) +} diff --git a/crates/codesmell/src/guide.rs b/crates/codesmell/src/guide.rs new file mode 100644 index 000000000..4570e524f --- /dev/null +++ b/crates/codesmell/src/guide.rs @@ -0,0 +1,115 @@ +//! Render the conventions pack an LLM reads before writing code, plus the +//! starter `policy.toml` template emitted by `codesmell init`. + +use crate::policy::Policy; +use crate::rhai::RhaiRuleLib; + +/// Human/LLM-readable conventions pack (doc §9). +/// +/// One line per enabled rule: `describe(params)` if the script defines it, +/// else the script's `ADVICE` constant, else a generic pointer to the entry. +/// This is the PREVENT side of the secure-vibe model (rules for the agent). +pub fn render_guide(policy: &Policy, lib: Option<&RhaiRuleLib>) -> String { + let mut lines = vec![ + "# Repository conventions (CodeSmell)".to_string(), + String::new(), + "Before writing or modifying code, follow these conventions:".to_string(), + String::new(), + ]; + let mut n: u32 = 1; + + match lib { + Some(lib) => { + for inst in lib.instances(policy) { + let line = lib + .describe(&inst) + .filter(|s| !s.is_empty()) + .unwrap_or_else(|| { + if inst.advice.is_empty() { + format!( + "Rule `{}` is enabled (see [[rhai.rule]] use = \"{}\").", + inst.rule_id, inst.use_script + ) + } else { + inst.advice.clone() + } + }); + lines.push(format!("{n}. {line}")); + n += 1; + } + } + None => { + for e in &policy.rhai.rules { + if !e.use_script.is_empty() { + lines.push(format!("{n}. Rule `{}` is enabled.", e.use_script)); + n += 1; + } + } + } + } + + if n == 1 { + lines.push("No team conventions are configured yet. Run `codesmell init` to start.".into()); + } + lines.join("\n") +} + +/// Starter `.codesmell/policy.toml` written by `codesmell init`. +pub const STARTER_POLICY: &str = r#"# CodeSmell policy — team engineering conventions. +# Run `codesmell guide` to print the conventions pack for the LLM. +# +# Every rule is a rhai script (builtin, or your own under .codesmell/rules/). +# `[[rhai.rule]]` enables + configures one; `params` is the script's input. + +version = 1 + +[rhai] +rule_dirs = [".codesmell/rules"] + +# --- style --- +[[rhai.rule]] +use = "style.function.max_lines" +params = { max = 60 } + +[[rhai.rule]] +use = "style.function.max_parameters" +params = { max = 4 } + +[[rhai.rule]] +use = "style.function.max_nesting" +params = { max = 4 } + +[[rhai.rule]] +use = "style.function.max_complexity" +params = { max = 10 } + +[[rhai.rule]] +use = "style.naming" +params = { rules = [{ kind = "method", pattern = "*Async", signature_contains = "async" }, { kind = "class", pattern = "*Service" }] } + +# --- architecture --- +[[rhai.rule]] +use = "architecture.boundary" +params = { layers = [{ name = "controller", paths = ["src/controllers/**", "**/*Controller.java"] }, { name = "service", paths = ["src/services/**", "**/*Service.java"] }, { name = "repository", paths = ["src/repositories/**", "**/*Repository.java"] }], deny = ["controller -> repository"] } + +# --- testing --- +[[rhai.rule]] +use = "testing.missing_test" +params = { require = true, test_paths = ["tests/**", "**/*_test.rs", "**/*_test.go", "**/test_*.py"], selectors = [{ layers = ["service"] }, { min_lines = 20 }] } + +# --- security (see `codesmell pack add security`) --- +# [[rhai.rule]] +# use = "security.deny_call" +# params = { deny = ["eval", "exec", "system"], message = "dynamic code execution" } + +[severity] # per-rule-id severity overrides +"style.function.max_lines" = "warning" +"architecture.boundary" = "blocking" +"testing.missing_test" = "required" +"#; + +/// Suggested AGENTS.md / CLAUDE.md snippet. +pub const AGENTS_SNIPPET: &str = r#"## CodeSmell conventions +Run `codesmell guide` before writing or modifying code, and `codesmell check` (or +`codesmell check --diff -`) after, then fix every violation by severity. +"#; diff --git a/crates/codesmell/src/index.rs b/crates/codesmell/src/index.rs new file mode 100644 index 000000000..2c902cf5c --- /dev/null +++ b/crates/codesmell/src/index.rs @@ -0,0 +1,19 @@ +//! Build an in-memory CodeGraph from a repository root. +//! +//! CodeSmell never relies on a pre-built `.codegraph` index: each run walks the +//! repo (honoring `.gitignore` via the extractor's walker) and parses it fresh +//! into a `GraphIndex::in_memory()`. CodeGraph stays the understanding layer; +//! only the persistent storage layer is dropped. + +use camino::Utf8Path; +use codegraph_extract::Orchestrator; +use codegraph_graph::GraphIndex; + +/// Parse `root` (whole repo) and return an in-memory graph. +pub async fn build_index(root: &Utf8Path) -> anyhow::Result { + let orch = Orchestrator::with_registry(); + let (parsed, _stats) = orch.parse_project(root)?; + let mut index = GraphIndex::in_memory(); + index.ingest(&parsed).await?; + Ok(index) +} diff --git a/crates/codesmell/src/lib.rs b/crates/codesmell/src/lib.rs new file mode 100644 index 000000000..acb5591c4 --- /dev/null +++ b/crates/codesmell/src/lib.rs @@ -0,0 +1,15 @@ +//! CodeSmell — team convention linter for maintainable, LLM-friendly code. +//! +//! CodeSmell consumes the code facts produced by CodeGraph (symbols, kinds, +//! dependencies, call graph) and evaluates them against a team's engineering +//! policy. It runs like a linter (`codesmell check`): the LLM reads the +//! conventions pack before writing code (`codesmell guide`) and fixes every +//! reported violation afterwards. + +pub mod engine; +pub mod glob; +pub mod guide; +pub mod index; +pub mod packs; +pub mod policy; +pub mod rhai; diff --git a/crates/codesmell/src/main.rs b/crates/codesmell/src/main.rs new file mode 100644 index 000000000..adf9d66f1 --- /dev/null +++ b/crates/codesmell/src/main.rs @@ -0,0 +1,307 @@ +//! CodeSmell CLI — a team convention linter (like eslint/clippy). + +use clap::{Parser, Subcommand, ValueEnum}; +use std::io::Read; +use std::path::PathBuf; + +use codegraph_graph::diff::parse_unified_diff; +use codesmell::engine::{evaluate, CheckScope}; +use codesmell::guide; +use codesmell::index::build_index; +use codesmell::packs; +use codesmell::packs::Registry; +use codesmell::policy; +use codesmell::rhai::RhaiRuleLib; + +#[derive(Parser)] +#[command( + name = "codesmell", + about = "Team convention linter for maintainable, LLM-friendly code" +)] +struct Cli { + /// Codesmell pack registry (path or git URL). Overrides CODESMELL_REGISTRY + /// and ~/.config/codesmell/config.toml. Used by `[[include]]` in policy.toml. + #[arg(long)] + registry: Option, + #[command(subcommand)] + command: Cmd, +} + +#[derive(Subcommand)] +enum Cmd { + /// Evaluate the policy over the repository (whole repo, paths, or a diff). + Check { + /// File or directory paths to scope the check to (default: whole repo). + paths: Vec, + /// Read a unified diff from a file path or `-` (stdin) and check only changed symbols. + #[arg(long)] + diff: Option, + /// Output format. + #[arg(long, value_enum, default_value = "human")] + format: OutFormat, + /// Exit non-zero when a violation of at least this severity is found. + #[arg(long, value_enum, default_value = "required")] + fail_on: policy::Severity, + }, + /// Print the conventions pack an LLM should read before writing code. + Guide { + /// Optional path to show conventions effective for that area. + path: Option, + }, + /// Write a starter `.codesmell/policy.toml` and print an AGENTS.md snippet. + Init { + /// Also install a built-in pack (e.g. `security`). + #[arg(long)] + pack: Option, + }, + /// Print the effective resolved policy as TOML. + Policy, + /// Manage built-in policy packs and the pack registry. + Pack { + #[command(subcommand)] + command: PackCmd, + }, +} + +#[derive(Subcommand)] +enum PackCmd { + /// List built-in packs and (if a registry is configured) registry packs. + List, + /// Copy a built-in pack's scripts + fragment into `.codesmell/` (idempotent). + Add { + /// Pack name from `codesmell pack list`. + name: String, + }, + /// Pre-fetch a registry pack into the local cache (no files copied to repo). + Pull { + /// Pack name from `codesmell pack list`. + name: String, + }, + /// Refresh the registry cache (git pull; no-op for local path registries). + Update, +} + +#[derive(Copy, Clone, ValueEnum)] +enum OutFormat { + Human, + Json, +} + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + let cli = Cli::parse(); + let cwd = std::env::current_dir()?; + let root = cwd.canonicalize().unwrap_or(cwd); + + match cli.command { + Cmd::Check { + paths, + diff, + format, + fail_on, + } => { + let registry = packs::resolve_registry(cli.registry.as_deref()).ok(); + check(&root, paths, diff, format, fail_on, registry.as_ref()).await + } + Cmd::Guide { path } => { + let registry = packs::resolve_registry(cli.registry.as_deref()).ok(); + guide(&root, path, registry.as_ref()) + } + Cmd::Init { pack } => init(&root, pack.as_deref()), + Cmd::Policy => { + let (p, _) = policy::load_policy(&root); + println!( + "{}", + toml::to_string_pretty(&p) + .unwrap_or_else(|_| "# (policy could not be serialized)".into()) + ); + Ok(()) + } + Cmd::Pack { command } => { + let registry = packs::resolve_registry(cli.registry.as_deref()); + pack(&root, command, registry) + } + } +} + +async fn check( + root: &std::path::Path, + paths: Vec, + diff: Option, + format: OutFormat, + fail_on: policy::Severity, + registry: Option<&Registry>, +) -> anyhow::Result<()> { + let (mut policy, found) = policy::load_policy(root); + if found.is_none() { + eprintln!( + "codesmell: no .codesmell/policy.toml found; using built-in defaults. \ + Run `codesmell init` to create one." + ); + } + packs::expand_includes(&mut policy, root, registry)?; + + let scope = if let Some(dp) = diff { + let text = if dp.as_os_str() == "-" { + let mut s = String::new(); + std::io::stdin().read_to_string(&mut s)?; + s + } else { + std::fs::read_to_string(&dp)? + }; + let parsed = + parse_unified_diff(&text).map_err(|e| anyhow::anyhow!("failed to parse diff: {e}"))?; + CheckScope::Diff(parsed) + } else if paths.is_empty() { + CheckScope::All + } else { + CheckScope::Paths( + paths + .iter() + .map(|p| p.to_string_lossy().into_owned()) + .collect(), + ) + }; + + let root_utf8 = camino::Utf8PathBuf::from_path_buf(root.to_path_buf()) + .map_err(|_| anyhow::anyhow!("non-UTF8 repository path"))?; + let index = build_index(&root_utf8).await?; + let report = evaluate(&index, &scope, &policy, root).await?; + + match format { + OutFormat::Human => print_human(&report), + OutFormat::Json => println!("{}", serde_json::to_string_pretty(&report)?), + } + + let should_fail = report.violations.iter().any(|v| v.severity >= fail_on); + if should_fail { + std::process::exit(1); + } + Ok(()) +} + +fn guide( + root: &std::path::Path, + path: Option, + registry: Option<&Registry>, +) -> anyhow::Result<()> { + let (mut p, _) = policy::load_policy(root); + packs::expand_includes(&mut p, root, registry)?; + let lib = RhaiRuleLib::load(root, &p.rhai.rule_dirs) + .map_err(|e| { + eprintln!("codesmell: warning: {e}"); + }) + .ok(); + if let Some(path) = path { + println!("# conventions effective for: {}", path.display()); + } + println!("{}", guide::render_guide(&p, lib.as_ref())); + Ok(()) +} + +fn print_human(report: &codesmell::engine::CheckReport) { + if report.violations.is_empty() { + println!("codesmell: no violations found."); + return; + } + for v in &report.violations { + println!("{}[{}]: {}", v.severity.as_label(), v.rule, v.message); + println!(" --> {}:{}", v.file, v.line); + println!(" hint: {}", v.fix_hint); + } + println!(); + let total: usize = report.summary.values().sum(); + let parts: Vec = report + .summary + .iter() + .map(|(k, n)| format!("{n} {k}")) + .collect(); + println!("{total} violation(s): {}", parts.join(", ")); +} + +fn init(root: &std::path::Path, pack: Option<&str>) -> anyhow::Result<()> { + let dir = root.join(".codesmell"); + std::fs::create_dir_all(&dir)?; + let path = dir.join("policy.toml"); + if path.exists() { + eprintln!( + "codesmell: {} already exists; not overwriting.", + path.display() + ); + } else { + std::fs::write(&path, guide::STARTER_POLICY)?; + println!("codesmell: wrote {}", path.display()); + } + println!("\nAdd this to AGENTS.md / CLAUDE.md so the LLM follows conventions:"); + println!("----"); + println!("{}", guide::AGENTS_SNIPPET); + + if let Some(name) = pack { + println!(); + match packs::builtin_packs().iter().find(|p| p.name == name) { + Some(pack) => packs::add_pack(root, pack)?, + None => { + eprintln!("codesmell: unknown pack `{name}`"); + eprintln!("available packs:"); + for p in packs::builtin_packs() { + eprintln!(" {}", p.name); + } + std::process::exit(1); + } + } + } + Ok(()) +} + +fn pack( + root: &std::path::Path, + command: PackCmd, + registry: anyhow::Result, +) -> anyhow::Result<()> { + match command { + PackCmd::List => { + println!("Built-in packs:"); + for p in packs::builtin_packs() { + println!(" {} — {}", p.name, p.description); + } + match ®istry { + Ok(reg) => { + println!("\nRegistry packs ({}):", reg.describe()); + for info in packs::list_registry_packs(reg)? { + let ver = info.version.map(|v| format!(" (v{v})")).unwrap_or_default(); + println!(" {} — {}{}", info.name, info.description, ver); + } + } + Err(_) => { + println!( + "\n(No registry configured; set --registry, CODESMELL_REGISTRY, or \ + ~/.config/codesmell/config.toml to list registry packs.)" + ); + } + } + Ok(()) + } + PackCmd::Add { name } => match packs::builtin_packs().iter().find(|p| p.name == name) { + Some(pack) => packs::add_pack(root, pack), + None => { + eprintln!("codesmell: unknown pack `{name}`"); + eprintln!("available packs:"); + for p in packs::builtin_packs() { + eprintln!(" {}", p.name); + } + std::process::exit(1); + } + }, + PackCmd::Pull { name } => { + let reg = registry?; + packs::pull_pack(®, &name)?; + Ok(()) + } + PackCmd::Update => { + let reg = registry?; + packs::update_registry(®)?; + println!("codesmell: registry updated."); + Ok(()) + } + } +} diff --git a/crates/codesmell/src/packs.rs b/crates/codesmell/src/packs.rs new file mode 100644 index 000000000..c79ccc025 --- /dev/null +++ b/crates/codesmell/src/packs.rs @@ -0,0 +1,415 @@ +//! Built-in policy packs, installed by `codesmell pack add`. +//! +//! A pack is a set of rule scripts plus a policy fragment (the `[[rhai.rule]]` +//! entries that enable + configure them). Both are embedded in the binary; +//! [`add_pack`] copies them into the repository — scripts into +//! `.codesmell/rules/` and the fragment into `.codesmell/packs/.policy.toml` +//! — so they can be edited or removed like any local config. + +use serde::Deserialize; +use std::path::{Path, PathBuf}; +use std::process::Command; + +use crate::policy::{merge_fragment_text, Policy}; +use anyhow::Context; + +/// A policy pack: rule scripts + a TOML fragment that enables them. +pub struct Pack { + pub name: &'static str, + pub description: &'static str, + /// TOML fragment to merge (appended to `.codesmell/packs/.policy.toml`). + pub fragment: &'static str, + /// `(file_name, source)` pairs copied into `.codesmell/rules/`. + pub rules: &'static [(&'static str, &'static str)], +} + +/// All built-in packs. +pub fn builtin_packs() -> &'static [Pack] { + &[SECURITY_PACK] +} + +/// Demo security pack: dangerous calls, weak crypto, unsafe deserialization. +/// Domain coverage is intentionally small to prove the mechanism; extend by +/// adding scripts + entries here. +pub const SECURITY_PACK: Pack = Pack { + name: "security", + description: "Dangerous calls, weak crypto, unsafe deserialization (demo pack).", + fragment: include_str!("../packs/security/policy.fragment.toml"), + rules: &[ + ( + "security.dangerous_exec.rhai", + include_str!("../packs/security/rules/security.dangerous_exec.rhai"), + ), + ( + "security.crypto_weak_hash.rhai", + include_str!("../packs/security/rules/security.crypto_weak_hash.rhai"), + ), + ( + "security.dangerous_deserialize.rhai", + include_str!("../packs/security/rules/security.dangerous_deserialize.rhai"), + ), + ], +}; + +/// Copy a pack's scripts + fragment into `root` (existing files are left +/// untouched so local edits are never silently overwritten). +pub fn add_pack(root: &Path, pack: &Pack) -> anyhow::Result<()> { + let rules_dir = root.join(".codesmell").join("rules"); + std::fs::create_dir_all(&rules_dir) + .with_context(|| format!("creating {}", rules_dir.display()))?; + for (file_name, src) in pack.rules { + let dest = rules_dir.join(file_name); + if dest.exists() { + eprintln!("codesmell: `{file_name}` already exists; not overwriting.",); + } else { + std::fs::write(&dest, src).with_context(|| format!("writing {}", dest.display()))?; + println!("codesmell: wrote {}", dest.display()); + } + } + + let packs_dir = root.join(".codesmell").join("packs"); + std::fs::create_dir_all(&packs_dir) + .with_context(|| format!("creating {}", packs_dir.display()))?; + let frag_dest = packs_dir.join(format!("{}.policy.toml", pack.name)); + if frag_dest.exists() { + eprintln!( + "codesmell: `{}.policy.toml` already exists; not overwriting.", + pack.name + ); + } else { + std::fs::write(&frag_dest, pack.fragment) + .with_context(|| format!("writing {}", frag_dest.display()))?; + println!("codesmell: wrote {}", frag_dest.display()); + } + println!( + "\nRun `codesmell policy` to see the merged policy, or edit the files under `.codesmell/`." + ); + Ok(()) +} + +// ==================== Registry (path or git URL, cached) ==================== + +/// A resolved registry source that holds pack directories. +#[derive(Clone)] +pub enum Registry { + /// A local directory containing one subdirectory per pack. + Path(PathBuf), + /// A git repository; packs are read from a per-machine clone in `cache`. + Git { url: String, cache: PathBuf }, +} + +impl Registry { + /// Human-readable source description for listings. + pub fn describe(&self) -> String { + match self { + Registry::Path(p) => p.display().to_string(), + Registry::Git { url, .. } => url.clone(), + } + } + + /// Build a local-path registry. + pub fn path(p: PathBuf) -> Self { + Registry::Path(p) + } + + /// Build a git registry with an explicit cache directory. + pub fn git(url: impl Into, cache: PathBuf) -> Self { + Registry::Git { + url: url.into(), + cache, + } + } +} + +/// Resolve the registry source (priority: explicit flag > `CODESMELL_REGISTRY` +/// env > `~/.config/codesmell/config.toml`). Errors if nothing is configured. +pub fn resolve_registry(explicit: Option<&str>) -> anyhow::Result { + let raw = explicit + .map(|s| s.to_string()) + .or_else(|| std::env::var("CODESMELL_REGISTRY").ok()) + .or_else(config_registry) + .ok_or_else(|| { + anyhow::anyhow!( + "no codesmell registry configured. Set --registry, the CODESMELL_REGISTRY \ + environment variable, or `registry = \"...\"` in ~/.config/codesmell/config.toml" + ) + })?; + classify_registry(raw.trim()) +} + +/// Like [`resolve_registry`] but returns `None` when no registry is configured +/// (used by check/guide, which only need a registry if a `[[include]]` actually +/// requires one). +pub fn resolve_registry_opt(explicit: Option<&str>) -> Option { + resolve_registry(explicit).ok() +} + +fn classify_registry(raw: &str) -> anyhow::Result { + if is_git_url(raw) { + Ok(Registry::Git { + url: raw.to_string(), + cache: registry_cache_dir(raw)?, + }) + } else { + Ok(Registry::Path(PathBuf::from(raw))) + } +} + +fn is_git_url(s: &str) -> bool { + s.starts_with("http://") + || s.starts_with("https://") + || s.starts_with("git@") + || s.starts_with("ssh://") + || s.starts_with("file://") + || s.contains("://") + || s.ends_with(".git") +} + +fn config_registry() -> Option { + let dir = dirs::config_dir()?; + let path = dir.join("codesmell").join("config.toml"); + let text = std::fs::read_to_string(path).ok()?; + let val: toml::Value = toml::from_str(&text).ok()?; + val.get("registry") + .and_then(|v| v.as_str()) + .map(str::to_string) +} + +fn registry_cache_dir(url: &str) -> anyhow::Result { + let cache = + dirs::cache_dir().ok_or_else(|| anyhow::anyhow!("cannot determine cache directory"))?; + let key: String = url + .chars() + .map(|c| { + if c.is_ascii_alphanumeric() || c == '.' || c == '-' || c == '_' { + c + } else { + '_' + } + }) + .collect(); + Ok(cache.join("codesmell").join("registry").join(key)) +} + +/// Resolve a pack's directory from a registry (cloning the git cache on first +/// use if needed). +pub fn resolve_pack_dir(registry: &Registry, name: &str) -> anyhow::Result { + match registry { + Registry::Path(base) => { + let dir = base.join(name); + if dir.is_dir() { + Ok(dir) + } else { + anyhow::bail!( + "pack `{name}` not found in registry path {}", + base.display() + ) + } + } + Registry::Git { url, cache } => { + ensure_cloned(url, cache)?; + let dir = cache.join(name); + if dir.is_dir() { + Ok(dir) + } else { + anyhow::bail!("pack `{name}` not found in registry {url}") + } + } + } +} + +fn ensure_cloned(url: &str, cache: &Path) -> anyhow::Result<()> { + if cache.join(".git").exists() { + return Ok(()); + } + if let Some(parent) = cache.parent() { + std::fs::create_dir_all(parent)?; + } + run_git(&[ + "clone".into(), + "--depth".into(), + "1".into(), + url.into(), + cache.to_string_lossy().into_owned(), + ]) +} + +fn run_git(args: &[String]) -> anyhow::Result<()> { + let status = Command::new("git") + .args(args) + .status() + .map_err(|e| anyhow::anyhow!("failed to run `git`: {e} (is git installed?)"))?; + if !status.success() { + anyhow::bail!("git {} failed", args.join(" ")); + } + Ok(()) +} + +/// Resolve every `[[include]]` in `policy`: merge each referenced pack's +/// fragment into the policy and register its `rules/` directory for the rhai +/// engine (appended to `policy.rhai.rule_dirs`). Pack files are NOT copied into +/// the repo — they are read directly from the (possibly cached) registry. +pub fn expand_includes( + policy: &mut Policy, + root: &Path, + registry: Option<&Registry>, +) -> anyhow::Result<()> { + if policy.includes.is_empty() { + return Ok(()); + } + let includes = std::mem::take(&mut policy.includes); + for inc in includes { + let pack_dir = if let Some(p) = &inc.path { + let dir = if Path::new(p).is_absolute() { + PathBuf::from(p) + } else { + root.join(p) + }; + if !dir.is_dir() { + anyhow::bail!("include path `{}` is not a directory", p); + } + dir + } else if let Some(name) = &inc.name { + let reg = match &inc.registry { + Some(r) => classify_registry(r.trim())?, + None => registry + .ok_or_else(|| { + anyhow::anyhow!( + "include `{name}` needs a registry; pass --registry, set CODESMELL_REGISTRY, \ + or add `registry = \"...\"` to ~/.config/codesmell/config.toml" + ) + })? + .clone(), + }; + resolve_pack_dir(®, name)? + } else { + anyhow::bail!("[[include]] needs `name` or `path`"); + }; + + let frag = pack_dir.join("policy.fragment.toml"); + if frag.exists() { + let text = std::fs::read_to_string(&frag).map_err(|e| { + anyhow::anyhow!("cannot read pack fragment {}: {e}", frag.display()) + })?; + merge_fragment_text(policy, &text); + } else { + eprintln!( + "codesmell: warning: include pack at {} has no policy.fragment.toml; skipping its rules", + pack_dir.display() + ); + } + + let rules = pack_dir.join("rules"); + if rules.is_dir() { + policy + .rhai + .rule_dirs + .push(rules.to_string_lossy().into_owned()); + } + } + Ok(()) +} + +/// Metadata for a pack available in a registry. +#[derive(Debug, Clone)] +pub struct PackInfo { + pub name: String, + pub description: String, + pub version: Option, +} + +#[derive(Debug, Deserialize)] +struct PackManifest { + name: Option, + description: Option, + version: Option, +} + +/// List packs available in a registry (cloning a git registry on first use). +pub fn list_registry_packs(registry: &Registry) -> anyhow::Result> { + let base = match registry { + Registry::Path(p) => p.clone(), + Registry::Git { url, cache } => { + ensure_cloned(url, cache)?; + cache.clone() + } + }; + let mut out = Vec::new(); + let Ok(entries) = std::fs::read_dir(&base) else { + return Ok(out); + }; + let mut dirs: Vec = entries + .flatten() + .map(|e| e.path()) + .filter(|p| p.is_dir()) + .collect(); + dirs.sort(); + for dir in dirs { + let name = dir + .file_name() + .and_then(|n| n.to_str()) + .unwrap_or("?") + .to_string(); + let mut info = PackInfo { + name: name.clone(), + description: "—".to_string(), + version: None, + }; + if let Some(meta) = read_pack_manifest(&dir) { + if let Some(n) = meta.name { + info.name = n; + } + if let Some(d) = meta.description { + info.description = d; + } + info.version = meta.version; + } + out.push(info); + } + Ok(out) +} + +fn read_pack_manifest(dir: &Path) -> Option { + let text = std::fs::read_to_string(dir.join("pack.toml")).ok()?; + toml::from_str::(&text).ok() +} + +/// Refresh a git registry (no-op for a local path registry). +pub fn update_registry(registry: &Registry) -> anyhow::Result<()> { + match registry { + Registry::Path(_) => { + println!("codesmell: registry is a local path; nothing to update."); + Ok(()) + } + Registry::Git { url, cache } => { + if cache.join(".git").exists() { + run_git(&[ + "-C".into(), + cache.to_string_lossy().into_owned(), + "pull".into(), + "--ff-only".into(), + ])?; + } else { + if let Some(parent) = cache.parent() { + std::fs::create_dir_all(parent)?; + } + run_git(&[ + "clone".into(), + "--depth".into(), + "1".into(), + url.clone(), + cache.to_string_lossy().into_owned(), + ])?; + } + Ok(()) + } + } +} + +/// Pre-fetch a pack into the local cache (useful for offline CI). No files are +/// copied into the repository; the pack stays in the registry cache. +pub fn pull_pack(registry: &Registry, name: &str) -> anyhow::Result { + let dir = resolve_pack_dir(registry, name)?; + println!("codesmell: pack `{}` available at {}", name, dir.display()); + Ok(dir) +} diff --git a/crates/codesmell/src/policy.rs b/crates/codesmell/src/policy.rs new file mode 100644 index 000000000..1c72d4445 --- /dev/null +++ b/crates/codesmell/src/policy.rs @@ -0,0 +1,347 @@ +//! Policy model + loading + pack-fragment merge. +//! +//! Policy lives in `.codesmell/policy.toml` (TOML, to match the codegraph +//! ecosystem). Every rule is a rhai script — builtin scripts ship inside the +//! binary, custom scripts live in rule dirs (default `.codesmell/rules/`). +//! A script only runs when an `[[rhai.rule]]` entry references it; `params` +//! parameterizes the script, so a script is a reusable rule template and the +//! policy file is its configuration. + +use clap::ValueEnum; +use serde::{Deserialize, Serialize}; +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +/// Severity of a violation, ordered least → most serious. +#[derive( + Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Serialize, Deserialize, Default, ValueEnum, +)] +#[serde(rename_all = "lowercase")] +pub enum Severity { + Info, + Warning, + #[default] + Required, + Blocking, +} + +impl Severity { + /// Human-facing label used in `warning[...]` / `error[...]` output. + pub fn as_label(self) -> &'static str { + match self { + Severity::Info => "note", + Severity::Warning => "warning", + Severity::Required => "error", + Severity::Blocking => "error", + } + } + + /// Parse a severity label (case-insensitive) — used by rhai rule results. + pub fn parse_label(s: &str) -> Option { + Some(match s.to_ascii_lowercase().as_str() { + "info" => Severity::Info, + "warning" => Severity::Warning, + "required" => Severity::Required, + "blocking" => Severity::Blocking, + _ => return None, + }) + } +} + +/// Ids of the rule scripts embedded in the binary (`rules_builtin/*.rhai`). +/// Kept as constants because tests, docs and the `[severity]` map refer to +/// them; a script's id is simply its file stem. +pub const RULE_MAX_LINES: &str = "style.function.max_lines"; +pub const RULE_MAX_PARAMS: &str = "style.function.max_parameters"; +pub const RULE_MAX_NESTING: &str = "style.function.max_nesting"; +pub const RULE_MAX_COMPLEXITY: &str = "style.function.max_complexity"; +pub const RULE_NAMING: &str = "style.naming"; +pub const RULE_BOUNDARY: &str = "architecture.boundary"; +pub const RULE_MISSING_TEST: &str = "testing.missing_test"; +pub const RULE_DENY_CALL: &str = "security.deny_call"; +pub const RULE_DENY_SYMBOL: &str = "security.deny_symbol"; + +// ==================== Rule entries ==================== + +/// One `[[rhai.rule]]` entry: enables + configures a rhai rule script. +/// +/// `use` names the script (its id / file stem — builtin or from a rule dir). +/// The same script may be referenced several times with different `params` +/// (e.g. a stricter limit for new code); `id` optionally renames the resulting +/// violations so entries can be told apart. +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(default)] +pub struct RuleEntry { + /// Script id to enable (builtin id or `.rhai` file stem). + #[serde(rename = "use")] + pub use_script: String, + /// Display rule id for violations (defaults to `use`). + pub id: Option, + /// Free-form parameters injected into the script as the `params` map. + pub params: Option, + /// Only apply to symbols whose file matches one of these globs + /// (repo-relative; empty = every file). + pub paths: Vec, + /// Exclude symbols whose file matches any of these globs (wins over + /// `paths`). + pub exclude: Vec, + /// Severity override for violations from this entry. + pub severity: Option, +} + +fn default_rule_dirs() -> Vec { + vec![".codesmell/rules".to_string()] +} + +/// The `[rhai]` section: where rule scripts are found + which are enabled. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct RhaiSection { + /// Directories (relative to the repo root) scanned for `*.rhai` rule + /// scripts. A user script with the same id as a builtin overrides it. + #[serde(default = "default_rule_dirs")] + pub rule_dirs: Vec, + /// Enabled rule entries; a script runs only when referenced here. + #[serde(default, rename = "rule")] + pub rules: Vec, +} + +impl Default for RhaiSection { + fn default() -> Self { + RhaiSection { + rule_dirs: default_rule_dirs(), + rules: Vec::new(), + } + } +} + +// ==================== Policy ==================== + +/// One `[[include]]` entry in `policy.toml`: pulls a pack into the policy by +/// reference. Its `policy.fragment.toml` is merged into the policy and its +/// `rules/` directory is scanned for rhai scripts — without copying the pack's +/// files into the repository. +/// +/// Set exactly one of `name` (resolved against a registry) or `path` (a local +/// pack directory, absolute or relative to the repo root). +#[derive(Debug, Clone, Default, Serialize, Deserialize)] +#[serde(default)] +pub struct IncludeEntry { + /// Pack name in the registry (resolved via `--registry` / `CODESMELL_REGISTRY` + /// / `~/.config/codesmell/config.toml`). + pub name: Option, + /// Direct path to a pack directory (absolute, or relative to the repo root). + pub path: Option, + /// Override the registry source for this include only. + pub registry: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(default)] +pub struct Policy { + pub version: u8, + #[serde(default)] + pub rhai: RhaiSection, + /// Per-rule-id severity overrides (win over each entry's default). + pub severity: HashMap, + /// Packs pulled in by reference (see [`IncludeEntry`]); serialized as the + /// `[[include]]` table in `policy.toml`. + #[serde(default, rename = "include")] + pub includes: Vec, +} + +impl Default for Policy { + fn default() -> Self { + Policy { + version: 1, + rhai: RhaiSection::default(), + severity: HashMap::new(), + includes: Vec::new(), + } + } +} + +impl Policy { + /// Severity for a rule id, falling back to the category default. + pub fn severity_of(&self, rule_id: &str) -> Severity { + self.severity + .get(rule_id) + .copied() + .unwrap_or_else(|| default_severity(rule_id)) + } +} + +/// Default severity per rule id (doc §11). Security rules fail the build by +/// default; everything else is a warning unless configured otherwise. +pub fn default_severity(rule_id: &str) -> Severity { + match rule_id { + RULE_BOUNDARY => Severity::Blocking, + RULE_MISSING_TEST => Severity::Required, + _ if rule_id.starts_with("security.") => Severity::Required, + _ => Severity::Warning, + } +} + +// ==================== Loading ==================== + +/// Load `.codesmell/policy.toml` by walking up from `start`, then merge every +/// `.codesmell/packs/*.policy.toml` fragment installed next to it (see +/// `codesmell pack add`). Fragment rule entries are appended and their +/// `[severity]` wins; scalars of the main policy are never overwritten. +/// +/// Returns `(policy, found_path)`; on missing/absent file a default policy is +/// returned (with `found_path = None`). +pub fn load_policy(start: &Path) -> (Policy, Option) { + let mut cur = Some(start.to_path_buf()); + while let Some(dir) = cur { + let candidate = dir.join(".codesmell").join("policy.toml"); + if candidate.exists() { + let mut policy = match std::fs::read_to_string(&candidate) { + Ok(text) => match toml::from_str::(&text) { + Ok(p) => p, + Err(e) => { + eprintln!( + "codesmell: warning: failed to parse {}: {e}; using defaults", + candidate.display() + ); + Policy::default() + } + }, + Err(e) => { + eprintln!( + "codesmell: warning: cannot read {}: {e}", + candidate.display() + ); + Policy::default() + } + }; + merge_pack_fragments(&mut policy, &dir.join(".codesmell").join("packs")); + return (policy, Some(candidate)); + } + cur = dir.parent().map(|p| p.to_path_buf()); + } + (Policy::default(), None) +} + +/// Merge a single pack fragment (TOML text) into `policy`: append its +/// `[[rhai.rule]]` entries and let its `[severity]` overrides win. A fragment +/// that fails to parse is reported loudly and skipped — a silently inactive +/// security pack is a hole. +pub(crate) fn merge_fragment_text(policy: &mut Policy, text: &str) -> bool { + match toml::from_str::(text) { + Ok(frag) => { + policy.rhai.rules.extend(frag.rhai.rules); + for (k, v) in frag.severity { + policy.severity.insert(k, v); + } + true + } + Err(e) => { + eprintln!("codesmell: warning: failed to parse pack fragment: {e}; fragment skipped"); + false + } + } +} + +/// Merge every `*.policy.toml` fragment under `packs_dir` into `policy` +/// (sorted by file name for determinism). +fn merge_pack_fragments(policy: &mut Policy, packs_dir: &Path) { + let Ok(entries) = std::fs::read_dir(packs_dir) else { + return; + }; + let mut files: Vec = entries + .flatten() + .map(|e| e.path()) + .filter(|p| { + p.file_name() + .and_then(|n| n.to_str()) + .is_some_and(|n| n.ends_with(".policy.toml")) + }) + .collect(); + files.sort(); + for file in files { + let text = match std::fs::read_to_string(&file) { + Ok(t) => t, + Err(e) => { + eprintln!( + "codesmell: warning: cannot read pack fragment {}: {e}", + file.display() + ); + continue; + } + }; + merge_fragment_text(policy, &text); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rule_entries_load_from_toml() { + let toml = r#" +version = 1 + +[[rhai.rule]] +use = "style.function.max_lines" +params = { max = 60 } + +[[rhai.rule]] +use = "style.function.max_lines" +id = "style.legacy_max_lines" +params = { max = 120 } +paths = ["legacy/**"] + +[[rhai.rule]] +use = "security.no_eval" +severity = "blocking" + +[severity] +"style.function.max_lines" = "info" +"#; + let p: Policy = toml::from_str(toml).unwrap(); + assert_eq!(p.rhai.rules.len(), 3); + assert_eq!(p.rhai.rules[0].use_script, "style.function.max_lines"); + assert_eq!(p.rhai.rule_dirs, vec![".codesmell/rules".to_string()]); + // entry severity override + assert_eq!(p.rhai.rules[2].severity, Some(Severity::Blocking)); + // [severity] map override + defaults + assert_eq!(p.severity_of(RULE_MAX_LINES), Severity::Info); + assert_eq!(p.severity_of(RULE_BOUNDARY), Severity::Blocking); + assert_eq!(p.severity_of("security.custom"), Severity::Required); + assert_eq!(p.severity_of(RULE_MAX_PARAMS), Severity::Warning); + } + + #[test] + fn custom_rule_dirs_are_respected() { + let p: Policy = toml::from_str("[rhai]\nrule_dirs = [\"policies\"]").unwrap(); + assert_eq!(p.rhai.rule_dirs, vec!["policies".to_string()]); + assert!(p.rhai.rules.is_empty()); + } + + #[test] + fn pack_fragments_append_rules_and_win_severity() { + let dir = tempfile::tempdir().unwrap(); + let packs = dir.path().join("packs"); + std::fs::create_dir_all(&packs).unwrap(); + std::fs::write( + packs.join("a.policy.toml"), + "[[rhai.rule]]\nuse = \"security.no_eval\"\n\n[severity]\n\"security.no_eval\" = \"blocking\"", + ) + .unwrap(); + let mut p = Policy::default(); + p.rhai.rules.push(RuleEntry { + use_script: RULE_MAX_LINES.into(), + ..Default::default() + }); + merge_pack_fragments(&mut p, &packs); + assert_eq!(p.rhai.rules.len(), 2); + assert_eq!(p.severity_of("security.no_eval"), Severity::Blocking); + + // A broken fragment is skipped loudly, not merged. + std::fs::write(packs.join("b.policy.toml"), "not [ valid toml").unwrap(); + let mut p2 = Policy::default(); + merge_pack_fragments(&mut p2, &packs); + assert_eq!(p2.rhai.rules.len(), 1); // only a.policy.toml's entry + } +} diff --git a/crates/codesmell/src/rhai.rs b/crates/codesmell/src/rhai.rs new file mode 100644 index 000000000..5ecaab84a --- /dev/null +++ b/crates/codesmell/src/rhai.rs @@ -0,0 +1,760 @@ +//! Rhai rule engine: load rule scripts, build configured instances, run them. +//! +//! Every CodeSmell rule is a rhai script. Builtin scripts ship inside the +//! binary (see [`builtin_scripts`]); custom scripts live in rule dirs +//! (default `.codesmell/rules/`) and override builtins of the same id. A script +//! only runs when an `[[rhai.rule]]` entry in the policy references it, and the +//! entry's `params` are injected as the script's `params` map (template +//! pattern). +//! +//! A script may define up to four hooks: +//! - `check(sym)` — for any symbol (function, method, class, variable, ...). +//! - `check_calls(sym, callees, callers)` — functions/methods only; `callees` +//! and `callers` are arrays of `{ name, file }` maps. +//! - `check_flow(sym, markers)` — functions/methods only; `markers` is the +//! ordered flow as an array of lowercase marker names (`"loop"`, `"if_true"`, +//! ...). +//! - `describe(params)` — optional; produces the human line for `codesmell guide`. +//! +//! A hook returns `false`/`()` to pass, a `string` for a default violation, or +//! a `{ message, hint, severity }` map for a full one. `const ADVICE` (set at +//! rule-definition time) documents the rule for the LLM guide. + +use codegraph_core::{marker_name, Symbol, SymbolKind}; +use codegraph_graph::GraphIndex; +use rhai::{Array, Dynamic, Engine, Map, Scope, AST}; +use std::cell::RefCell; +use std::collections::HashMap; +use std::path::{Path, PathBuf}; + +use crate::engine::{collect_symbols, rel_path, CheckScope, Violation}; +use crate::glob::GlobSet; +use crate::policy::{Policy, Severity}; + +const CHECK: &str = "check"; +const CHECK_CALLS: &str = "check_calls"; +const CHECK_FLOW: &str = "check_flow"; +const DESCRIBE: &str = "describe"; + +/// Symbol kinds handed to `check` (code-like decls; parameters, modules, files +/// and config are excluded as noise). +pub const RULE_SYMBOL_KINDS: &[SymbolKind] = &[ + SymbolKind::Function, + SymbolKind::Method, + SymbolKind::Class, + SymbolKind::Interface, + SymbolKind::Enum, + SymbolKind::Variable, + SymbolKind::Constant, + SymbolKind::Field, +]; + +/// Symbol kinds for `check_calls` / `check_flow` (only symbols with a call graph). +const FN_KINDS: &[SymbolKind] = &[SymbolKind::Function, SymbolKind::Method]; + +#[derive(Default, Clone, Copy)] +struct HookSet { + check: bool, + check_calls: bool, + check_flow: bool, + describe: bool, +} + +/// A compiled rule script (by id). +pub struct RhaiRule { + ast: AST, + hooks: HookSet, + /// `const ADVICE` captured at load. + advice: String, +} + +/// One enabled, configured rule: a compiled script + the policy entry's params, +/// scoping and severity. +pub struct RuleInstance { + pub rule_id: String, + pub use_script: String, + pub advice: String, + params: Map, + paths: GlobSet, + exclude: GlobSet, + pub severity: Option, + hooks: HookSet, +} + +/// Loaded set of rule scripts (builtins + user dirs) sharing one engine with +/// the `glob` / `regex_match` helper functions registered. +pub struct RhaiRuleLib { + engine: Engine, + rules: HashMap, +} + +impl RhaiRuleLib { + /// Load builtin scripts plus every `*.rhai` under `dirs` (relative to + /// `root`); a user script overrides a builtin of the same id. A script that + /// fails to compile aborts the whole lint — a silently inactive rule is a + /// security hole, not a warning. + pub fn load(root: &Path, dirs: &[String]) -> anyhow::Result { + let engine = build_engine(); + let mut rules: HashMap = HashMap::new(); + let mut errors: Vec = Vec::new(); + + for (id, src) in builtin_scripts() { + match compile_rule(&engine, id, src) { + Ok(r) => { + rules.insert(id.to_string(), r); + } + Err(e) => errors.push(format!("builtin rule `{id}`: {e}")), + } + } + for dir in dirs { + let abs = if Path::new(dir).is_absolute() { + PathBuf::from(dir) + } else { + root.join(dir) + }; + let Ok(entries) = std::fs::read_dir(&abs) else { + continue; + }; + for entry in entries.flatten() { + let path = entry.path(); + if path.extension().and_then(|x| x.to_str()) != Some("rhai") { + continue; + } + let Some(id) = path + .file_stem() + .and_then(|s| s.to_str()) + .map(str::to_string) + else { + continue; + }; + let Ok(src) = std::fs::read_to_string(&path) else { + continue; + }; + match compile_rule(&engine, &id, &src) { + Ok(r) => { + rules.insert(id.clone(), r); + } + Err(e) => errors.push(format!("{}: {e}", path.display())), + } + } + } + + if !errors.is_empty() { + anyhow::bail!("failed to compile rule script(s):\n{}", errors.join("\n")); + } + Ok(RhaiRuleLib { engine, rules }) + } + + /// Build the enabled instances from the policy. Entries whose `use` does + /// not resolve to a loaded script are reported loudly (not silently) so a + /// typo'd security rule cannot hide. + pub fn instances(&self, policy: &Policy) -> Vec { + let mut out = Vec::new(); + for e in &policy.rhai.rules { + if e.use_script.is_empty() { + eprintln!("codesmell: warning: [[rhai.rule]] with empty `use` — skipped"); + continue; + } + match self.rules.get(&e.use_script) { + None => eprintln!( + "codesmell: warning: [[rhai.rule]] use = \"{}\" — no such rule in rule dirs; skipped", + e.use_script + ), + Some(r) => { + let params = e.params.as_ref().map(toml_to_map).unwrap_or_default(); + out.push(RuleInstance { + rule_id: e.id.clone().unwrap_or_else(|| e.use_script.clone()), + use_script: e.use_script.clone(), + advice: r.advice.clone(), + params, + paths: GlobSet::new(&e.paths), + exclude: GlobSet::new(&e.exclude), + severity: e.severity, + hooks: r.hooks, + }); + } + } + } + out + } + + /// Human-facing description for `codesmell guide` (from `describe(params)`), + /// if the script defines one; `None` falls back to `advice`. + pub fn describe(&self, inst: &RuleInstance) -> Option { + if !inst.hooks.describe { + return None; + } + let mut scope = Scope::new(); + self.engine + .call_fn::( + &mut scope, + &self.rules[&inst.use_script].ast, + DESCRIBE, + (inst.params.clone(),), + ) + .ok() + } + + /// Evaluate `instance`'s `hook` with `args` (each arg is a `Dynamic`). + /// Returns `None` to pass (or on a runtime error, reported to stderr); + /// otherwise the returned value the script flagged with. + pub(crate) fn invoke( + &self, + inst: &RuleInstance, + hook: &str, + label: &str, + args: impl rhai::FuncArgs, + ) -> Option { + let mut scope = Scope::new(); + scope.push_constant("params", inst.params.clone()); + match self.engine.call_fn::( + &mut scope, + &self.rules[&inst.use_script].ast, + hook, + args, + ) { + Ok(d) => { + if d.is_unit() || matches!(d.clone().try_cast::(), Some(false)) { + None + } else { + Some(d) + } + } + Err(e) => { + eprintln!( + "codesmell: warning: rule `{}` failed on `{}`: {e}", + inst.rule_id, label + ); + None + } + } + } +} + +// ==================== Engine + helpers ==================== + +fn build_engine() -> Engine { + let mut engine = Engine::new(); + // Map literals use rhai's `#{ ... }` syntax; keep expression nesting unlimited + // so realistic rule bodies (long string concatenations, nested `if`) compile. + engine.set_max_expr_depths(0, 0); + engine.register_fn("glob", glob_match); + engine.register_fn("regex_match", regex_match); + engine +} + +thread_local! { + static RX: RefCell>> = RefCell::new(HashMap::new()); +} + +/// `regex_match(pattern, text)` — cached per pattern. +fn regex_match(pattern: &str, text: &str) -> bool { + RX.with(|c| { + let mut cache = c.borrow_mut(); + let re = cache + .entry(pattern.to_string()) + .or_insert_with(|| regex::Regex::new(pattern).ok()); + re.as_ref().map(|r| r.is_match(text)).unwrap_or(false) + }) +} + +/// `glob(pattern, text)` — glob match (wraps the crate `glob` helper). +fn glob_match(pattern: &str, text: &str) -> bool { + crate::glob::glob_matches(pattern, text) +} + +fn compile_rule(engine: &Engine, _id: &str, src: &str) -> anyhow::Result { + let ast = engine.compile(src).map_err(|e| anyhow::anyhow!("{e}"))?; + // Run top-level statements so `const ADVICE` is defined in the scope; + // function definitions are collected into the AST library regardless. + let mut scope = Scope::new(); + engine + .run_ast_with_scope(&mut scope, &ast) + .map_err(|e| anyhow::anyhow!("{e}"))?; + let advice = scope + .get("ADVICE") + .and_then(|d| d.clone().try_cast::()) + .unwrap_or_default(); + let mut hooks = HookSet::default(); + for f in ast.iter_functions() { + #[allow(clippy::borrow_deref_ref)] + match &*f.name { + CHECK => hooks.check = true, + CHECK_CALLS => hooks.check_calls = true, + CHECK_FLOW => hooks.check_flow = true, + DESCRIBE => hooks.describe = true, + _ => {} + } + } + Ok(RhaiRule { ast, hooks, advice }) +} + +// ==================== Symbol → rhai value conversion ==================== + +/// Build the `sym` map passed to `check` hooks. `pub(crate)` so tests can craft +/// synthetic symbols without a real index. +pub(crate) fn symbol_map(s: &Symbol, root: &Path) -> Map { + let mut m = Map::new(); + m.insert("name".into(), Dynamic::from(s.name.clone())); + m.insert("kind".into(), Dynamic::from(s.kind.as_str().to_string())); + m.insert("scope".into(), Dynamic::from(s.scope.as_str().to_string())); + m.insert("file".into(), Dynamic::from(rel_path(&s.file, root))); + m.insert("line".into(), Dynamic::from(s.line as i64)); + m.insert("end_line".into(), Dynamic::from(s.end_line as i64)); + m.insert( + "signature".into(), + Dynamic::from(s.signature.clone().unwrap_or_default()), + ); + m.insert( + "doc".into(), + Dynamic::from(s.doc.clone().unwrap_or_default()), + ); + m.insert("language".into(), Dynamic::from(s.language.clone())); + let anns: Array = s + .annotations + .iter() + .map(|a| { + let mut am = Map::new(); + am.insert("name".into(), Dynamic::from(a.name.clone())); + am.insert("line".into(), Dynamic::from(a.line as i64)); + Dynamic::from(am) + }) + .collect(); + m.insert("annotations".into(), Dynamic::from(anns)); + m +} + +fn callee_maps(callees: &[Symbol], root: &Path) -> Array { + callees + .iter() + .map(|c| { + let mut m = Map::new(); + m.insert("name".into(), Dynamic::from(c.name.clone())); + m.insert("file".into(), Dynamic::from(rel_path(&c.file, root))); + Dynamic::from(m) + }) + .collect() +} + +/// Lowercase marker names from a flow chain — the `markers` array passed to +/// `check_flow`. +fn marker_names(chain: &[u64]) -> Array { + chain + .iter() + .filter_map(|&id| marker_name(id).map(|n| Dynamic::from(n.to_lowercase()))) + .collect() +} + +// ==================== Result interpretation ==================== + +/// Interpret a flagged value into `(message, hint, severity-from-map)`. +fn interpret_result( + d: &Dynamic, + sym_name: &str, + rule_id: &str, +) -> (String, String, Option) { + if let Some(s) = d.clone().try_cast::() { + return (s, default_hint(sym_name, rule_id), None); + } + if let Some(m) = d.clone().try_cast::() { + let msg = map_str(&m, "message"); + let hint = map_str(&m, "hint"); + let sev = m + .get("severity") + .and_then(|v| v.clone().try_cast::()) + .and_then(|s| Severity::parse_label(&s)); + let msg = if msg.is_empty() { + format!("rule `{rule_id}` flagged `{sym_name}`") + } else { + msg + }; + return ( + msg, + if hint.is_empty() { + default_hint(sym_name, rule_id) + } else { + hint + }, + sev, + ); + } + ( + format!("rule `{rule_id}` flagged `{sym_name}`"), + default_hint(sym_name, rule_id), + None, + ) +} + +fn default_hint(sym_name: &str, rule_id: &str) -> String { + format!("adjust `{sym_name}` to satisfy rule `{rule_id}`") +} + +fn map_str(m: &Map, k: &str) -> String { + m.get(k) + .and_then(|v| v.clone().try_cast::()) + .unwrap_or_default() +} + +// ==================== Run ==================== + +/// Evaluate every enabled rule instance over the repository and return the +/// violations. Returns an empty report (and no work) when the policy enables no +/// rules. +pub async fn run( + index: &GraphIndex, + scope: &CheckScope, + policy: &Policy, + root: &Path, +) -> anyhow::Result> { + if policy.rhai.rules.is_empty() { + return Ok(Vec::new()); + } + let lib = RhaiRuleLib::load(root, &policy.rhai.rule_dirs)?; + let insts = lib.instances(policy); + if insts.is_empty() { + return Ok(Vec::new()); + } + + let needs_all = insts.iter().any(|i| i.hooks.check); + let needs_calls = insts.iter().any(|i| i.hooks.check_calls); + let needs_flow = insts.iter().any(|i| i.hooks.check_flow); + + let mut violations: Vec = Vec::new(); + + if needs_all { + for s in collect_symbols(index, RULE_SYMBOL_KINDS, scope, root) { + let sym = symbol_map(&s, root); + for inst in &insts { + if !inst.hooks.check || !in_scope(inst, &s, root) { + continue; + } + if let Some(d) = lib.invoke(inst, CHECK, &s.name, (sym.clone(),)) { + emit(&mut violations, inst, &s, policy, &d); + } + } + } + } + + if needs_calls || needs_flow { + for s in collect_symbols(index, FN_KINDS, scope, root) { + let mut callees: Option> = None; + let mut callers: Option> = None; + let mut markers: Option = None; + for inst in &insts { + if !in_scope(inst, &s, root) { + continue; + } + if inst.hooks.check_calls { + if callees.is_none() { + callees = Some(index.callees(s.id).await.unwrap_or_default()); + } + if callers.is_none() { + callers = Some(index.callers(s.id, 1).await.unwrap_or_default()); + } + let args = ( + symbol_map(&s, root), + callee_maps(callees.as_ref().unwrap(), root), + callee_maps(callers.as_ref().unwrap(), root), + ); + if let Some(d) = lib.invoke(inst, CHECK_CALLS, &s.name, args) { + emit(&mut violations, inst, &s, policy, &d); + } + } + if inst.hooks.check_flow { + if markers.is_none() { + markers = Some(match index.flow(s.id).await { + Ok(f) => marker_names(&f.chain), + Err(_) => Vec::new(), + }); + } + let args = (symbol_map(&s, root), markers.clone().unwrap()); + if let Some(d) = lib.invoke(inst, CHECK_FLOW, &s.name, args) { + emit(&mut violations, inst, &s, policy, &d); + } + } + } + } + } + + Ok(violations) +} + +fn in_scope(inst: &RuleInstance, s: &Symbol, root: &Path) -> bool { + let rel = rel_path(&s.file, root); + if inst.exclude.matches(&rel) { + return false; + } + inst.paths.is_empty() || inst.paths.matches(&rel) +} + +fn emit( + violations: &mut Vec, + inst: &RuleInstance, + s: &Symbol, + policy: &Policy, + d: &Dynamic, +) { + let (message, hint, result_sev) = interpret_result(d, &s.name, &inst.rule_id); + let severity = result_sev + .or(inst.severity) + .unwrap_or_else(|| policy.severity_of(&inst.rule_id)); + violations.push(Violation { + rule: inst.rule_id.clone(), + severity, + file: s.file.clone(), + line: s.line, + symbol: s.name.clone(), + message, + fix_hint: hint, + }); +} + +// ==================== toml::Value → rhai Map ==================== + +fn toml_to_map(v: &toml::Value) -> Map { + let mut m = Map::new(); + if let Some(t) = v.as_table() { + for (k, val) in t { + m.insert(k.clone().into(), toml_to_dynamic(val)); + } + } + m +} + +fn toml_to_dynamic(v: &toml::Value) -> Dynamic { + match v { + toml::Value::String(s) => Dynamic::from(s.clone()), + toml::Value::Integer(i) => Dynamic::from(*i), + toml::Value::Float(f) => Dynamic::from(*f), + toml::Value::Boolean(b) => Dynamic::from(*b), + toml::Value::Array(a) => Dynamic::from(a.iter().map(toml_to_dynamic).collect::()), + toml::Value::Table(t) => Dynamic::from(toml_to_map_table(t)), + _ => Dynamic::from(()), + } +} + +fn toml_to_map_table(t: &toml::map::Map) -> Map { + let mut m = Map::new(); + for (k, val) in t { + m.insert(k.clone().into(), toml_to_dynamic(val)); + } + m +} + +// ==================== Builtin rule scripts ==================== + +/// Builtin rule scripts shipped in the binary (rule templates). A user script +/// with the same id overrides these. Ids equal the legacy rule ids so the +/// `[severity]` map and existing docs keep working. +pub fn builtin_scripts() -> &'static [(&'static str, &'static str)] { + &[ + ( + crate::policy::RULE_MAX_LINES, + include_str!("../rules_builtin/style.function.max_lines.rhai"), + ), + ( + crate::policy::RULE_MAX_PARAMS, + include_str!("../rules_builtin/style.function.max_parameters.rhai"), + ), + ( + crate::policy::RULE_MAX_NESTING, + include_str!("../rules_builtin/style.function.max_nesting.rhai"), + ), + ( + crate::policy::RULE_MAX_COMPLEXITY, + include_str!("../rules_builtin/style.function.max_complexity.rhai"), + ), + ( + crate::policy::RULE_NAMING, + include_str!("../rules_builtin/style.naming.rhai"), + ), + ( + crate::policy::RULE_BOUNDARY, + include_str!("../rules_builtin/architecture.boundary.rhai"), + ), + ( + crate::policy::RULE_MISSING_TEST, + include_str!("../rules_builtin/testing.missing_test.rhai"), + ), + ( + crate::policy::RULE_DENY_CALL, + include_str!("../rules_builtin/security.deny_call.rhai"), + ), + ( + crate::policy::RULE_DENY_SYMBOL, + include_str!("../rules_builtin/security.deny_symbol.rhai"), + ), + ] +} + +#[allow(dead_code)] +fn _assert_path_exists() { + // Compile-time guard that the builtin script files are reachable. + const _: &str = include_str!("../rules_builtin/style.function.max_lines.rhai"); +} + +/// Test helper: build a minimal `RuleInstance` for a builtin script with params. +#[cfg(test)] +pub(crate) fn test_instance(use_script: &str, params: Map) -> RuleInstance { + let lib = RhaiRuleLib::load(Path::new("."), &[]).unwrap(); + let r = lib.rules.get(use_script).expect("builtin script exists"); + RuleInstance { + rule_id: use_script.to_string(), + use_script: use_script.to_string(), + advice: r.advice.clone(), + params, + paths: GlobSet::new(&[]), + exclude: GlobSet::new(&[]), + severity: None, + hooks: r.hooks, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::policy::{RULE_DENY_CALL, RULE_DENY_SYMBOL, RULE_MAX_LINES, RULE_MAX_PARAMS}; + use codegraph_core::ScopeLevel; + + fn fake_sym(name: &str, kind: SymbolKind, line: u32, end_line: u32, sig: &str) -> Symbol { + Symbol { + id: 0, + name: name.to_string(), + kind, + scope: ScopeLevel::Global, + scope_id: 0, + type_ref: 0, + type_name: None, + file: "src/x.rs".to_string(), + line, + end_line, + signature: Some(sig.to_string()), + doc: None, + annotations: vec![], + language: "rust".to_string(), + } + } + + fn param_map(entries: &[(&str, i64)]) -> Map { + let mut m = Map::new(); + for (k, v) in entries { + m.insert((*k).into(), Dynamic::from(*v)); + } + m + } + + fn arr(items: &[&str]) -> Array { + items.iter().map(|s| Dynamic::from(s.to_string())).collect() + } + + #[test] + fn builtin_scripts_compile_and_are_unique() { + let lib = RhaiRuleLib::load(Path::new("."), &[]).unwrap(); + assert_eq!(builtin_scripts().len(), lib.rules.len()); + let ids: std::collections::HashSet<&str> = + builtin_scripts().iter().map(|(id, _)| *id).collect(); + assert_eq!(ids.len(), builtin_scripts().len(), "duplicate builtin ids"); + } + + #[test] + fn max_lines_flags_over_limit() { + let inst = test_instance(RULE_MAX_LINES, param_map(&[("max", 5)])); + let lib = RhaiRuleLib::load(Path::new("."), &[]).unwrap(); + let s = fake_sym("big", SymbolKind::Function, 1, 10, "fn big() {}"); + let sym = symbol_map(&s, Path::new(".")); + let d = lib.invoke(&inst, CHECK, &s.name, (sym,)).unwrap(); + let (msg, _, _) = interpret_result(&d, &s.name, RULE_MAX_LINES); + assert!(msg.contains("lines"), "msg was: {msg}"); + } + + #[test] + fn max_parameters_counts_real_params_and_skips_self() { + let inst = test_instance(RULE_MAX_PARAMS, param_map(&[("max", 3)])); + let lib = RhaiRuleLib::load(Path::new("."), &[]).unwrap(); + // 4 real params (self excluded) → violation; nested parens in a default value must not miscount + let s = fake_sym( + "f", + SymbolKind::Method, + 1, + 2, + "pub fn f(&self, a: i32, b: i32, c: Option<(i32, i32)>, d: i32) -> i32", + ); + let sym = symbol_map(&s, Path::new(".")); + let d = lib.invoke(&inst, CHECK, &s.name, (sym,)).unwrap(); + let (msg, _, _) = interpret_result(&d, &s.name, RULE_MAX_PARAMS); + assert!(msg.contains("parameters"), "msg was: {msg}"); + } + + #[test] + fn params_default_via_null_coalesce() { + // No max in params → script default (60) applies; a 5-line fn passes. + let inst = test_instance(RULE_MAX_LINES, Map::new()); + let lib = RhaiRuleLib::load(Path::new("."), &[]).unwrap(); + let s = fake_sym("small", SymbolKind::Function, 1, 5, "fn small() {}"); + let sym = symbol_map(&s, Path::new(".")); + assert!(lib.invoke(&inst, CHECK, &s.name, (sym,)).is_none()); + } + + #[test] + fn deny_call_flags_dangerous_sink() { + let mut params = Map::new(); + params.insert("deny".into(), Dynamic::from(arr(&["eval"]))); + let inst = test_instance(RULE_DENY_CALL, params); + let lib = RhaiRuleLib::load(Path::new("."), &[]).unwrap(); + let s = fake_sym("run", SymbolKind::Function, 1, 2, "fn run() {}"); + let callees = vec![fake_sym("eval", SymbolKind::Function, 1, 1, "fn eval() {}")]; + let args = ( + symbol_map(&s, Path::new(".")), + callee_maps(&callees, Path::new(".")), + Array::new(), + ); + let d = lib.invoke(&inst, CHECK_CALLS, &s.name, args).unwrap(); + let (msg, _, _) = interpret_result(&d, &s.name, RULE_DENY_CALL); + assert!(msg.contains("eval"), "msg was: {msg}"); + } + + #[test] + fn deny_symbol_flags_secret_const() { + let mut params = Map::new(); + params.insert("kind".into(), Dynamic::from("constant".to_string())); + params.insert( + "name_re".into(), + Dynamic::from(arr(&["(?i)^(PASSWORD|SECRET)$"])), + ); + let inst = test_instance(RULE_DENY_SYMBOL, params); + let lib = RhaiRuleLib::load(Path::new("."), &[]).unwrap(); + let s = fake_sym("PASSWORD", SymbolKind::Constant, 1, 1, "PASSWORD = \"x\""); + let sym = symbol_map(&s, Path::new(".")); + let d = lib.invoke(&inst, CHECK, &s.name, (sym,)).unwrap(); + let (msg, _, _) = interpret_result(&d, &s.name, RULE_DENY_SYMBOL); + assert!(msg.contains("PASSWORD"), "msg was: {msg}"); + } + + #[test] + fn deny_call_symbol_filter_uses_glob_helper() { + // script uses `glob()` to match the caller name; "do_eval" matches *Async? no, + // but it proves the glob helper is reachable inside a script. + let mut params = Map::new(); + params.insert("deny".into(), Dynamic::from(arr(&["eval"]))); + params.insert("symbols".into(), Dynamic::from(arr(&["do_*"]))); // glob filter + let inst = test_instance(RULE_DENY_CALL, params); + let lib = RhaiRuleLib::load(Path::new("."), &[]).unwrap(); + // caller "do_eval" matches symbols glob → flagged + let s = fake_sym("do_eval", SymbolKind::Function, 1, 2, "fn do_eval() {}"); + let callees = vec![fake_sym("eval", SymbolKind::Function, 1, 1, "fn eval() {}")]; + let args = ( + symbol_map(&s, Path::new(".")), + callee_maps(&callees, Path::new(".")), + Array::new(), + ); + assert!(lib.invoke(&inst, CHECK_CALLS, &s.name, args).is_some()); + // caller "other" does NOT match symbols glob → not flagged + let s2 = fake_sym("other", SymbolKind::Function, 1, 2, "fn other() {}"); + let args2 = ( + symbol_map(&s2, Path::new(".")), + callee_maps(&callees, Path::new(".")), + Array::new(), + ); + assert!(lib.invoke(&inst, CHECK_CALLS, &s2.name, args2).is_none()); + } +} diff --git a/crates/codesmell/tests/engine_tests.rs b/crates/codesmell/tests/engine_tests.rs new file mode 100644 index 000000000..6cfde95fc --- /dev/null +++ b/crates/codesmell/tests/engine_tests.rs @@ -0,0 +1,327 @@ +//! End-to-end tests: build an in-memory CodeGraph from a fixture and evaluate +//! the policy. Fixtures live under `tests/fixtures/`. + +use codegraph_graph::diff::parse_unified_diff; +use codesmell::engine::{evaluate, CheckScope}; +use codesmell::index::build_index; +use codesmell::packs::{self, Registry, SECURITY_PACK}; +use codesmell::policy; +use codesmell::rhai::RhaiRuleLib; +use std::collections::HashSet; +use std::path::{Path, PathBuf}; +use std::process::Command; + +async fn index_for(name: &str) -> (PathBuf, codegraph_graph::GraphIndex) { + let root = Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures") + .join(name); + let root_utf8 = camino::Utf8PathBuf::from_path_buf(root.clone()).unwrap(); + let idx = build_index(&root_utf8).await.unwrap(); + (root, idx) +} + +fn rules_for(report: &codesmell::engine::CheckReport, symbol: &str) -> HashSet { + report + .violations + .iter() + .filter(|v| v.symbol == symbol) + .map(|v| v.rule.clone()) + .collect() +} + +#[tokio::test] +async fn rustshop_flags_every_rule_category() { + let (root, idx) = index_for("rustshop").await; + let (p, _) = policy::load_policy(&root); + let report = evaluate(&idx, &CheckScope::All, &p, &root).await.unwrap(); + + let place = rules_for(&report, "place_order"); + assert!( + place.contains("architecture.boundary"), + "expected boundary violation, got {place:?}" + ); + assert!( + place.contains("style.function.max_parameters"), + "expected max_parameters violation, got {place:?}" + ); + assert!( + place.contains("style.naming"), + "expected naming violation, got {place:?}" + ); + + assert!(rules_for(&report, "compute_big").contains("style.function.max_lines")); + assert!(rules_for(&report, "unreached_logic").contains("testing.missing_test")); +} + +#[tokio::test] +async fn rustshop_diff_scope_narrows_to_changed_file() { + let (root, idx) = index_for("rustshop").await; + let (p, _) = policy::load_policy(&root); + + // A hunk covering only `compute_big` (lines 6..41) of price_service.rs. + let diff = "\ +diff --git a/src/services/price_service.rs b/src/services/price_service.rs +--- a/src/services/price_service.rs ++++ b/src/services/price_service.rs +@@ -6,36 +6,36 @@ impl PriceService { ++// edited +"; + let parsed = parse_unified_diff(diff).unwrap(); + let report = evaluate(&idx, &CheckScope::Diff(parsed), &p, &root) + .await + .unwrap(); + + // Only `compute_big` overlaps the hunk; the controller's issues are out of scope. + assert_eq!(report.violations.len(), 1, "got {:?}", report.violations); + assert_eq!(report.violations[0].symbol, "compute_big"); + assert_eq!(report.violations[0].rule, "style.function.max_lines"); +} + +#[tokio::test] +async fn cleanshop_has_no_violations() { + let (root, idx) = index_for("cleanshop").await; + let (p, _) = policy::load_policy(&root); + let report = evaluate(&idx, &CheckScope::All, &p, &root).await.unwrap(); + assert!( + report.violations.is_empty(), + "expected zero violations, got {:?}", + report.violations + ); +} + +#[tokio::test] +async fn secshop_flags_declarative_and_custom_rules() { + let (root, idx) = index_for("secshop").await; + let (p, _) = policy::load_policy(&root); + let report = evaluate(&idx, &CheckScope::All, &p, &root).await.unwrap(); + + // declarative deny_symbol (constant name pattern) + assert!( + rules_for(&report, "API_KEY").contains("security.deny_symbol"), + "expected security.deny_symbol on API_KEY, got {:?}", + rules_for(&report, "API_KEY") + ); + // declarative deny_call (denied callee) + assert!( + rules_for(&report, "run_script").contains("security.deny_call"), + "expected security.deny_call on run_script, got {:?}", + rules_for(&report, "run_script") + ); + // team-authored rhai rule template (custom rule) + assert!( + rules_for(&report, "may_panic").contains("team.no_panic"), + "expected team.no_panic on may_panic, got {:?}", + rules_for(&report, "may_panic") + ); +} + +#[test] +fn pack_install_copies_files_and_is_idempotent_and_merges() { + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + + // No policy yet, just install the pack. + packs::add_pack(root, &SECURITY_PACK).unwrap(); + let rule = root.join(".codesmell/rules/security.dangerous_exec.rhai"); + let frag = root.join(".codesmell/packs/security.policy.toml"); + assert!(rule.exists(), "pack script not installed"); + assert!(frag.exists(), "pack fragment not installed"); + + // Idempotent: second install must not overwrite (and must not error). + let before = std::fs::read(&rule).unwrap(); + packs::add_pack(root, &SECURITY_PACK).unwrap(); + assert_eq!( + std::fs::read(&rule).unwrap(), + before, + "pack install overwrote a file" + ); + + // With a minimal main policy, the fragment is merged and the pack scripts compile. + std::fs::write( + root.join(".codesmell/policy.toml"), + "[rhai]\nrule_dirs = [\".codesmell/rules\"]\n", + ) + .unwrap(); + let (p, found) = policy::load_policy(root); + assert!(found.is_some(), "policy not found after pack install"); + let uses_security: Vec<&str> = p.rhai.rules.iter().map(|r| r.use_script.as_str()).collect(); + assert!( + uses_security + .iter() + .any(|u| *u == "security.dangerous_exec"), + "pack fragment rules not merged: {uses_security:?}" + ); + + // The pack's rhai scripts must compile as a rule library. + let lib = RhaiRuleLib::load(root, &p.rhai.rule_dirs); + assert!( + lib.is_ok(), + "pack scripts failed to compile: {:?}", + lib.err() + ); +} + +// A minimal rhai rule used by the include/registry tests. +const DEMO_RULE: &str = r#" +const ADVICE = "demo rule for testing includes"; +fn check(sym) { + if sym.name == "smell_me" { + "found smell_me" + } +} +"#; + +/// `[[include]] path = ""` pulls a pack's fragment + rules into the +/// policy without copying files into the repo. +#[test] +fn include_local_path_pulls_pack_rules() { + let reg = tempfile::tempdir().unwrap(); + let pack = reg.path().join("demo"); + std::fs::create_dir_all(pack.join("rules")).unwrap(); + std::fs::write( + pack.join("policy.fragment.toml"), + "[[rhai.rule]]\nuse = \"demo.smell\"\n", + ) + .unwrap(); + std::fs::write(pack.join("rules/demo.smell.rhai"), DEMO_RULE).unwrap(); + + let proj = tempfile::tempdir().unwrap(); + std::fs::create_dir_all(proj.path().join(".codesmell")).unwrap(); + std::fs::write( + proj.path().join(".codesmell/policy.toml"), + format!( + "version = 1\n\n[[include]]\npath = \"{}\"\n", + pack.display() + ), + ) + .unwrap(); + + let (mut policy, found) = policy::load_policy(proj.path()); + assert!(found.is_some(), "policy not found"); + packs::expand_includes(&mut policy, proj.path(), None).unwrap(); + + assert!( + policy + .rhai + .rules + .iter() + .any(|r| r.use_script == "demo.smell"), + "pack fragment not merged into policy" + ); + let lib = RhaiRuleLib::load(proj.path(), &policy.rhai.rule_dirs) + .expect("rule library should compile"); + assert!( + lib.instances(&policy) + .iter() + .any(|i| i.rule_id == "demo.smell"), + "pack rule not instantiated" + ); +} + +/// A `[[include]]` with `name` but no configured registry must error loudly. +#[test] +fn include_name_without_registry_errors() { + let mut p = policy::Policy::default(); + p.includes.push(policy::IncludeEntry { + name: Some("x".into()), + ..Default::default() + }); + let err = packs::expand_includes(&mut p, Path::new("."), None); + assert!(err.is_err(), "expected error when registry is missing"); +} + +/// A git registry (here a local `file://` repo) is cloned into the cache by +/// `update_registry`, then resolved offline by `[[include]] name = ...`. +#[test] +fn include_from_git_registry_resolves_offline_after_update() { + if Command::new("git").arg("--version").status().is_err() { + eprintln!("git not available; skipping git registry test"); + return; + } + + let work = tempfile::tempdir().unwrap(); + let src = work.path().join("srcrepo"); + std::fs::create_dir_all(src.join("demo/rules")).unwrap(); + std::fs::write( + src.join("demo/policy.fragment.toml"), + "[[rhai.rule]]\nuse = \"demo.smell\"\n", + ) + .unwrap(); + std::fs::write(src.join("demo/rules/demo.smell.rhai"), DEMO_RULE).unwrap(); + std::fs::write( + src.join("demo/pack.toml"), + "name = \"demo\"\ndescription = \"demo pack\"\nversion = \"0.1.0\"\n", + ) + .unwrap(); + + let git = |args: &[&str]| { + let status = Command::new("git") + .current_dir(&src) + .args(args) + .status() + .expect("git runs"); + assert!(status.success(), "git {:?} failed", args); + }; + git(&["init", "-q"]); + git(&[ + "-c", + "user.email=test@test", + "-c", + "user.name=test", + "add", + "-A", + ]); + git(&[ + "-c", + "user.email=test@test", + "-c", + "user.name=test", + "commit", + "-q", + "-m", + "add pack", + ]); + + let url = format!("file://{}", src.canonicalize().unwrap().display()); + let cache = work.path().join("cache"); + let reg = Registry::git(url, cache.clone()); + + packs::update_registry(®).unwrap(); + assert!(cache.join("demo").is_dir(), "pack not cloned into cache"); + + let proj = work.path().join("proj"); + std::fs::create_dir_all(proj.join(".codesmell")).unwrap(); + std::fs::write( + proj.join(".codesmell/policy.toml"), + "version = 1\n\n[[include]]\nname = \"demo\"\n", + ) + .unwrap(); + + let (mut policy, found) = policy::load_policy(&proj); + assert!(found.is_some()); + packs::expand_includes(&mut policy, &proj, Some(®)).unwrap(); + + assert!( + policy + .rhai + .rules + .iter() + .any(|r| r.use_script == "demo.smell"), + "pack fragment not merged from git registry" + ); + let lib = RhaiRuleLib::load(&proj, &policy.rhai.rule_dirs).expect("rule lib compiles"); + assert!( + lib.instances(&policy) + .iter() + .any(|i| i.rule_id == "demo.smell"), + "pack rule not instantiated from git registry" + ); + + let infos = packs::list_registry_packs(®).unwrap(); + assert!( + infos + .iter() + .any(|p| p.name == "demo" && p.version.as_deref() == Some("0.1.0")), + "pack metadata not listed: {infos:?}" + ); +} diff --git a/crates/codesmell/tests/fixtures/cleanshop/.codesmell/policy.toml b/crates/codesmell/tests/fixtures/cleanshop/.codesmell/policy.toml new file mode 100644 index 000000000..802283fb7 --- /dev/null +++ b/crates/codesmell/tests/fixtures/cleanshop/.codesmell/policy.toml @@ -0,0 +1,31 @@ +version = 1 + +[style.function] +max_lines = 30 +max_parameters = 4 +max_nesting = 4 + +[[style.naming.rule]] +kind = "method" +pattern = "*Async" +signature_contains = "async" + +[[architecture.layer]] +name = "controller" +paths = ["src/controllers/**"] + +[[architecture.layer]] +name = "service" +paths = ["src/services/**"] + +[[architecture.layer]] +name = "repository" +paths = ["src/repositories/**"] + +[[architecture.boundary]] +deny = ["controller -> repository"] + +[testing] +require_tests_for_changed_logic = true +test_paths = ["tests/**", "**/test_*.rs", "**/*_test.rs"] +logic_selectors = [{ layers = ["service"] }, { min_lines = 20 }] diff --git a/crates/codesmell/tests/fixtures/cleanshop/src/controllers/order_controller.rs b/crates/codesmell/tests/fixtures/cleanshop/src/controllers/order_controller.rs new file mode 100644 index 000000000..9870aa9e7 --- /dev/null +++ b/crates/codesmell/tests/fixtures/cleanshop/src/controllers/order_controller.rs @@ -0,0 +1,7 @@ +pub struct OrderController; + +impl OrderController { + pub fn place_order(&self, svc: &OrderService, id: i32) -> i32 { + svc.place(id) + } +} diff --git a/crates/codesmell/tests/fixtures/cleanshop/src/services/good_service.rs b/crates/codesmell/tests/fixtures/cleanshop/src/services/good_service.rs new file mode 100644 index 000000000..8445f7d91 --- /dev/null +++ b/crates/codesmell/tests/fixtures/cleanshop/src/services/good_service.rs @@ -0,0 +1,5 @@ +// Clean service code: short, few parameters, no denied boundaries, and tested. + +pub fn calc_total(a: i32, b: i32) -> i32 { + a + b +} diff --git a/crates/codesmell/tests/fixtures/cleanshop/src/services/order_service.rs b/crates/codesmell/tests/fixtures/cleanshop/src/services/order_service.rs new file mode 100644 index 000000000..33411ae3a --- /dev/null +++ b/crates/codesmell/tests/fixtures/cleanshop/src/services/order_service.rs @@ -0,0 +1,7 @@ +pub struct OrderService; + +impl OrderService { + pub fn place(&self, id: i32) -> i32 { + id + } +} diff --git a/crates/codesmell/tests/fixtures/cleanshop/tests/order_service_test.rs b/crates/codesmell/tests/fixtures/cleanshop/tests/order_service_test.rs new file mode 100644 index 000000000..86e628729 --- /dev/null +++ b/crates/codesmell/tests/fixtures/cleanshop/tests/order_service_test.rs @@ -0,0 +1,5 @@ +#[test] +fn place_is_covered() { + let s = OrderService; + let _ = s.place(1); +} diff --git a/crates/codesmell/tests/fixtures/cleanshop/tests/test_good_service.rs b/crates/codesmell/tests/fixtures/cleanshop/tests/test_good_service.rs new file mode 100644 index 000000000..ab09da3cc --- /dev/null +++ b/crates/codesmell/tests/fixtures/cleanshop/tests/test_good_service.rs @@ -0,0 +1,4 @@ +#[test] +fn calc_total_is_covered() { + assert_eq!(calc_total(1, 2), 3); +} diff --git a/crates/codesmell/tests/fixtures/rustshop/.codesmell/policy.toml b/crates/codesmell/tests/fixtures/rustshop/.codesmell/policy.toml new file mode 100644 index 000000000..bc28d63e9 --- /dev/null +++ b/crates/codesmell/tests/fixtures/rustshop/.codesmell/policy.toml @@ -0,0 +1,39 @@ +version = 1 + +[rhai] +rule_dirs = [".codesmell/rules"] + +# --- style --- +[[rhai.rule]] +use = "style.function.max_lines" +params = { max = 30 } + +[[rhai.rule]] +use = "style.function.max_parameters" +params = { max = 4 } + +[[rhai.rule]] +use = "style.function.max_nesting" +params = { max = 4 } + +[[rhai.rule]] +use = "style.function.max_complexity" +params = { max = 10 } + +[[rhai.rule]] +use = "style.naming" +params = { rules = [{ kind = "method", pattern = "*Async", signature_contains = "async" }] } + +# --- architecture --- +[[rhai.rule]] +use = "architecture.boundary" +params = { layers = [{ name = "controller", paths = ["src/controllers/**"] }, { name = "service", paths = ["src/services/**"] }, { name = "repository", paths = ["src/repositories/**"] }], deny = ["controller -> repository"] } + +# --- testing --- +[[rhai.rule]] +use = "testing.missing_test" +params = { require = true, test_paths = ["tests/**", "**/test_*.rs", "**/*_test.rs"], selectors = [{ layers = ["service"] }, { min_lines = 20 }] } + +[severity] +"architecture.boundary" = "blocking" +"testing.missing_test" = "required" diff --git a/crates/codesmell/tests/fixtures/rustshop/src/controllers/order_controller.rs b/crates/codesmell/tests/fixtures/rustshop/src/controllers/order_controller.rs new file mode 100644 index 000000000..b37d3ba5c --- /dev/null +++ b/crates/codesmell/tests/fixtures/rustshop/src/controllers/order_controller.rs @@ -0,0 +1,21 @@ +pub struct OrderController; + +impl OrderController { + // async method not named *Async + too many parameters (6 > 4) + + // calls the repository directly (controller -> repository denied). + pub async fn place_order( + &self, + repo: &OrderRepo, + a: i32, + b: i32, + c: i32, + d: i32, + e: i32, + f: i32, + ) -> i32 { + repo.save_order(1); + repo.save_order(2); + repo.save_order(3); + 0 + } +} diff --git a/crates/codesmell/tests/fixtures/rustshop/src/repositories/order_repo.rs b/crates/codesmell/tests/fixtures/rustshop/src/repositories/order_repo.rs new file mode 100644 index 000000000..64bdb0320 --- /dev/null +++ b/crates/codesmell/tests/fixtures/rustshop/src/repositories/order_repo.rs @@ -0,0 +1,7 @@ +pub struct OrderRepo; + +impl OrderRepo { + pub fn save_order(&self, id: i32) -> i32 { + id + } +} diff --git a/crates/codesmell/tests/fixtures/rustshop/src/services/price_service.rs b/crates/codesmell/tests/fixtures/rustshop/src/services/price_service.rs new file mode 100644 index 000000000..9ac4bbb48 --- /dev/null +++ b/crates/codesmell/tests/fixtures/rustshop/src/services/price_service.rs @@ -0,0 +1,69 @@ +pub struct PriceService; + +impl PriceService { + // Long business-logic method (>30 lines) with a caller in tests, + // so it is exercised but also flagged for length. + pub fn compute_big(&self, base: i32) -> i32 { + let mut total = base; + total += 1; + total += 2; + total += 3; + total += 4; + total += 5; + total += 6; + total += 7; + total += 8; + total += 9; + total += 10; + total += 11; + total += 12; + total += 13; + total += 14; + total += 15; + total += 16; + total += 17; + total += 18; + total += 19; + total += 20; + total += 21; + total += 22; + total += 23; + total += 24; + total += 25; + total += 26; + total += 27; + total += 28; + total += 29; + total += 30; + total += 31; + total += 32; + total + } + + // Business logic (>20 lines) with NO test reference -> missing test. + pub fn unreached_logic(&self, base: i32) -> i32 { + let mut total = base; + total += 1; + total += 2; + total += 3; + total += 4; + total += 5; + total += 6; + total += 7; + total += 8; + total += 9; + total += 10; + total += 11; + total += 12; + total += 13; + total += 14; + total += 15; + total += 16; + total += 17; + total += 18; + total += 19; + total += 20; + total += 21; + total + } +} diff --git a/crates/codesmell/tests/fixtures/rustshop/tests/test_price_service.rs b/crates/codesmell/tests/fixtures/rustshop/tests/test_price_service.rs new file mode 100644 index 000000000..4db2b17cd --- /dev/null +++ b/crates/codesmell/tests/fixtures/rustshop/tests/test_price_service.rs @@ -0,0 +1,10 @@ +// Integration test that exercises `compute_big` so it is considered "tested" +// (and therefore not flagged by testing.missing_test). `unreached_logic` has no +// such caller and is expected to be flagged. + +#[test] +fn compute_big_is_covered() { + let svc = PriceService {}; + let got = svc.compute_big(3); + assert_eq!(got, 3 + 55); +} diff --git a/crates/codesmell/tests/fixtures/secshop/.codesmell/policy.toml b/crates/codesmell/tests/fixtures/secshop/.codesmell/policy.toml new file mode 100644 index 000000000..22522c599 --- /dev/null +++ b/crates/codesmell/tests/fixtures/secshop/.codesmell/policy.toml @@ -0,0 +1,18 @@ +version = 1 + +[rhai] +rule_dirs = [".codesmell/rules"] + +# declarative-style security rules expressed as rhai rule templates +[[rhai.rule]] +use = "security.deny_call" +params = { deny = ["eval_cli"], message = "dynamic code execution" } + +[[rhai.rule]] +use = "security.deny_symbol" +params = { kind = "constant", name_re = ['(?i)^(API_KEY|SECRET)$'], message = "hard-coded secret" } + +# a team-authored rule template shipped in .codesmell/rules/ +[[rhai.rule]] +use = "team.no_panic" +params = { max_panics = 0 } diff --git a/crates/codesmell/tests/fixtures/secshop/.codesmell/rules/team.no_panic.rhai b/crates/codesmell/tests/fixtures/secshop/.codesmell/rules/team.no_panic.rhai new file mode 100644 index 000000000..ef20db3f6 --- /dev/null +++ b/crates/codesmell/tests/fixtures/secshop/.codesmell/rules/team.no_panic.rhai @@ -0,0 +1,23 @@ +// team.no_panic — library code should not panic. +// A reusable rule template: `params.max_panics` lets a team allow a few panics +// in legacy code while forbidding them in new code. +const ADVICE = "Avoid panics; return a Result instead."; + +fn describe(params) { + "Functions should not panic (limit " + (params.max_panics ?? 0).to_string() + ")." +} + +fn check_calls(sym, callees, callers) { + let max = params.max_panics ?? 0; + let hits = callees.filter(|c| { + let l = c.name.to_lower(); + l == "panic" || l.contains("panic") + }); + if hits.len() > max { + return #{ + message: "`" + sym.name + "` can panic via `" + hits[0].name + "`", + hint: "return a Result and propagate the error" + }; + } + false +} diff --git a/crates/codesmell/tests/fixtures/secshop/src/lib.rs b/crates/codesmell/tests/fixtures/secshop/src/lib.rs new file mode 100644 index 000000000..9ae7a792d --- /dev/null +++ b/crates/codesmell/tests/fixtures/secshop/src/lib.rs @@ -0,0 +1,21 @@ +// Security fixture: a hard-coded secret constant, a call to a denied sink, and a +// function that can panic. + +pub const API_KEY: &str = "do-not-commit-me"; + +pub fn run_script(code: &str) -> i32 { + eval_cli(code) +} + +fn eval_cli(_code: &str) -> i32 { + 0 +} + +pub fn may_panic(x: i32) -> i32 { + if x < 0 { + do_panic(); + } + x +} + +fn do_panic() {} diff --git a/docs/PLAN.md b/docs/PLAN.md index 10318aeaf..d0b64ee98 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -1,71 +1,53 @@ -# CodeGraph — Rust Rewrite Plan - -Port intégral du projet TS (`archive/`) vers Rust natif. Objectif: binaire `<15MB` stripped (vs 140MB Node bundle), parse 2-5× plus rapide, zero runtime dep. - -## Non-objectifs - -- Pas de compatibilité DB avec `archive/.codegraph/`. Schema repart neuf. -- Pas de wrapper npm. Distribution = `cargo install` + binaires GitHub Releases. -- Pas de port 1:1 du code TS. On reproduit le comportement observable (NodeKind, EdgeKind, surface MCP, CLI), pas la structure interne. - -## Architecture cible - -``` -crates/ - codegraph-core/ types + erreurs (NodeKind, EdgeKind, Node, Edge) - codegraph-db/ rusqlite + schema + prepared stmts + FTS5 - codegraph-extract/ tree-sitter natif + extractors par langage - codegraph-resolve/ imports, name-match, frameworks - codegraph-graph/ traversal (callers/callees/impact) - codegraph-context/ builder markdown/json - codegraph-mcp/ stdio JSON-RPC 2.0 hand-rolled - codegraph-installer/ 5 cibles agents (claude/cursor/codex/opencode/hermes) - codegraph/ binaire CLI (clap) + watcher (notify) -``` - -Pipeline runtime: -``` -files → ignore-walker → parse-workers (rayon, tree-sitter) → batch DB tx - ↓ - ReferenceResolver (imports + frameworks) - ↓ - GraphTraverser ← ContextBuilder - ↓ - MCP server / CLI commands -``` - -## Ordre d'implémentation - -| # | Étape | Spec | Dépend de | -|---|---|---|---| -| 1 | Bootstrap workspace | [01-bootstrap.md](specs/01-bootstrap.md) | — | -| 2 | Core types | [02-core-types.md](specs/02-core-types.md) | 1 | -| 3 | DB layer | [03-db-layer.md](specs/03-db-layer.md) | 2 | -| 4 | Extraction + langages | [04-extraction.md](specs/04-extraction.md) | 3 | -| 5 | Résolution + frameworks | [05-resolution.md](specs/05-resolution.md) | 4 | -| 6 | Graph + context | [06-graph-context.md](specs/06-graph-context.md) | 3 | -| 7 | MCP server | [07-mcp-server.md](specs/07-mcp-server.md) | 6 | -| 8 | Installer | [08-installer.md](specs/08-installer.md) | 1 | -| 9 | CLI + watcher | [09-cli-watcher.md](specs/09-cli-watcher.md) | 4,5,6,7,8 | -| 10 | Release pipeline | [10-release.md](specs/10-release.md) | 9 | - -Étapes 1-2 done. Étape 6 peut paralléliser avec 4-5 (utilise seulement DB read). -Étape 8 indépendante du reste (pure file ops). - -## Cibles binaire - -- `cargo build --release`: profil `release` (LTO fat, codegen-units=1, strip, panic=abort) -- Estimation: ~12MB Linux x86_64 stripped avec 15 grammaires tree-sitter statiques + SQLite bundled -- Si dépasse 20MB: profil `release-small` (`opt-level=z`) + features off pour langages exotiques - -## Tests - -- Unit tests in-crate avec `#[cfg(test)]` -- Integration tests dans `crates/*/tests/` -- Fixtures synthétiques par langage dans `tests/fixtures/` -- Pas de DB mock — tempdir + rusqlite réel (cf archive/__tests__) -- Eval harness reporté post-MVP (équivalent `__tests__/evaluation/`) - -## Suivi - -État des tâches dans TaskList runtime. Cette doc + specs sont source de vérité pour le quoi/pourquoi. +# CodeGraph — documentation index & roadmap + +All documentation lives in `docs/`. The README is the landing page; everything +detailed is here. Each spec records the *what/why* of a shipped component — +when behavior changes, update the spec with it. + +## Documentation map + +| Document | Audience | Covers | +|---|---|---| +| [codegraph.md](codegraph.md) | Agents | Usage guide **embedded into the binary** as MCP server instructions: session binding, tool selection by intent, timeout/resume, response formats, sandbox/diff contracts | +| [configuration.md](configuration.md) | Users / ops | `.codegraph/config.toml` reference: languages, `[[effect_rules]]` + defaults, storage backends & sharding, `[sandbox]`, `[embedding]`, ignore files, CLI flags, watcher | +| [mcp.md](mcp.md) | Users / ops | Running the MCP server (stdio + Streamable HTTP), client configuration per agent, the 27-tool catalog, common arguments, token conventions | +| [sandbox.md](sandbox.md) | Users / agents | Behavior sandbox: `[sandbox]` config, the Rhai mock contract with examples, run semantics, `codegraph_sandbox` / `codegraph_diff_simulate` / `codegraph_origin_simulate` | +| [codesmell.md](codesmell.md) | Users / agents | The CodeSmell team-convention linter: `.codesmell/policy.toml` schema, rules, CLI, agent workflow | +| [benchmarks/storage-perf.md](benchmarks/storage-perf.md) | Devs | Storage benchmark snapshot (in-memory vs sqlite vs lmdb) | +| [specs/](specs/) | Devs | Per-component design specs, kept in sync with the code | + +## Specs + +| # | Spec | Status | +|---|---|---| +| 01 | [Workspace bootstrap](specs/01-bootstrap.md) — 11-crate layout, conventions | done | +| 02 | [Core types](specs/02-core-types.md) — semgraph model (Symbol, chains, markers, effects) | done | +| 03 | [Storage layer](specs/03-db-layer.md) — `Storage` trait, sqlite/lmdb/redis/postgres/mysql, sharding | done | +| 04 | [Extraction](specs/04-extraction.md) — tree-sitter, declarative `LangSpec`, walker, effects | done | +| 05 | [Call resolution](specs/05-resolution.md) — ingest resolve phases, scoring, call-name index | done | +| 06 | [GraphIndex & context](specs/06-graph-context.md) — engines, queries, GraphApi, diff engine | done | +| 07 | [MCP server](specs/07-mcp-server.md) — rmcp, stdio + HTTP, 27 tools, token conventions | done | +| 08 | [Installer](specs/08-installer.md) — multi-agent client setup, idempotence | done | +| 09 | [CLI & watcher](specs/09-cli-watcher.md) — init/deinit/embed/serve, debounced full re-index | done | +| 10 | [Release & CI](specs/10-release.md) — CI jobs, distribution, features, publish order | done | +| 11 | [CodeSmell](specs/11-codesmell.md) — convention linter over in-memory CodeGraph facts | done (MVP) | + +## Roadmap / fast-follow + +- **CodeSmell**: convention discovery (statistics → candidate policies with + confidence), coverage threshold enforcement, policy history/evolution, + runtime/engineering policies (timeouts/retries via effect classification). +- **MCP HTTP hardening**: enforce `--api-key` bearer auth, mount `/health` / + `/metrics` observability endpoints (flags currently accepted but inert). +- **Watcher**: honor `.codegraphignore` (today only `.gitignore` is + consulted for event filtering). +- **Embeddings**: make `codegraph-api`'s unconditional feature pull-in + optional so slim builds can drop the ONNX runtime. + +## History + +The Rust rewrite plan that originally lived here (Node/Edge model, +`codegraph-db`, `codegraph-resolve`, hand-rolled JSON-RPC, <15 MB target) is +superseded — each deviation is recorded in the "Deviations" section of the +relevant spec. The rewrite itself completed: single ~58 MB binary with every +backend bundled, semgraph model, 27-tool MCP server on the rmcp SDK. diff --git a/crates/codegraph-bench/STORAGE_PERF.md b/docs/benchmarks/storage-perf.md similarity index 100% rename from crates/codegraph-bench/STORAGE_PERF.md rename to docs/benchmarks/storage-perf.md diff --git a/crates/codegraph-mcp/src/server-instructions.md b/docs/codegraph.md similarity index 98% rename from crates/codegraph-mcp/src/server-instructions.md rename to docs/codegraph.md index 5c6940367..eb4a96504 100644 --- a/crates/codegraph-mcp/src/server-instructions.md +++ b/docs/codegraph.md @@ -76,7 +76,7 @@ finds every `*Service` class), and `exact`. Use `total` + `offset` to page. ## Large indexes: timeout + resume -On very large indexes a broad search (`codegraph_search`, `codegraph_search_symbol`, +On very large indexes a broad search (`codegraph_search_symbol`, `codegraph_search_by_annotation`, `codegraph_search_flow`, `codegraph_references`, `codegraph_search_by_call`, `codegraph_list_classes`, `codegraph_list_interfaces`) can exceed its time budget. All of these tools accept `timeout_ms` (default `20000`; @@ -105,8 +105,6 @@ times out yields a fresh resume id. response includes a `resume` id in addition to `total`/`has_more` — pass it on the next call (with a new `offset`) to page further **without re-scanning** the index. -- `codegraph_search` on success returns a plain array (no `resume` field); if - you need more results, narrow the query or use `codegraph_search_symbol`. ## Trust the results @@ -115,7 +113,7 @@ that's slower, less accurate, and wastes context. ## Output detail & token usage -Symbols in list-tool responses (`codegraph_search`, `codegraph_callers`, +Symbols in list-tool responses (`codegraph_callers`, `codegraph_callees`, `codegraph_impact`, `codegraph_search_symbol`, `codegraph_search_by_annotation`, `codegraph_list_classes`, `codegraph_list_interfaces`, and the symbol embedded in `codegraph_flow`) are diff --git a/docs/codesmell.md b/docs/codesmell.md new file mode 100644 index 000000000..a42ec0bab --- /dev/null +++ b/docs/codesmell.md @@ -0,0 +1,173 @@ +# CodeSmell — team convention linter + +CodeSmell is a team-specific engineering convention and code-quality policy +engine. It runs like a linter (`codesmell check`) and answers one question: + +> Does this code look and behave like code that this team would normally write +> and maintain? + +It is designed for both developers and LLM coding agents: the agent reads the +conventions pack **before** writing code (`codesmell guide`) and fixes every +reported violation **after** (`codesmell check`). Each violation carries a +`fix_hint`, so the LLM repairs code instead of guessing. + +CodeSmell does not re-implement code analysis — it consumes the facts produced +by CodeGraph (symbols, kinds, line spans, call graph, call sites). Each run +parses the repository fresh into an in-memory CodeGraph, so there is no index +to maintain and no staleness: just run it. + +```text +Code → CodeGraph (in-memory) → Facts → CodeSmell → Violations + fix hints +``` + +## Install / build + +```bash +cargo build -p codesmell # binary at target/debug/codesmell +cargo install --path crates/codesmell +``` + +## Quick start + +```bash +codesmell init # write .codesmell/policy.toml (commented starter) +codesmell guide # print the conventions pack for the LLM / team +codesmell check # lint the whole repository +codesmell check src/services # lint a subtree +git diff | codesmell check --diff - # lint only changed symbols +codesmell policy # print the effective resolved policy +``` + +Suggested `AGENTS.md` / `CLAUDE.md` snippet (also printed by `codesmell init`): + +```markdown +## CodeSmell conventions +Run `codesmell guide` before writing or modifying code, and `codesmell check` +(or `codesmell check --diff -`) after, then fix every violation by severity. +``` + +## Policy file + +`.codesmell/policy.toml` (TOML; loaded by walking up from the current +directory). Every section is optional — an empty policy simply checks nothing. + +```toml +version = 1 + +[style.function] +max_lines = 60 # function body length (end_line - line + 1) +max_parameters = 4 # real parameters parsed from the signature (self excluded) +max_nesting = 4 # best-effort depth from control-flow markers + +[[style.naming.rule]] +kind = "class" # SymbolKind: function/method/class/... +pattern = "*Service" # glob matched against the symbol name +paths = ["src/services/**"] # optional scoping + +[[style.naming.rule]] +kind = "method" +pattern = "*Async" # async methods must end in Async +signature_contains = "async" # condition: only applies to async declarations + +[[architecture.layer]] +name = "controller" +paths = ["src/controllers/**", "**/*Controller.java"] + +[[architecture.layer]] +name = "repository" +paths = ["src/repositories/**", "**/*Repository.java"] + +[[architecture.boundary]] +deny = ["controller -> repository"] # enforced +allow = ["controller -> service"] # informational in MVP + +[testing] +require_tests_for_changed_logic = true +test_paths = ["tests/**", "**/*_test.go", "**/*_test.rs", "**/test_*.py", "**/*Test.java"] +logic_selectors = [{ layers = ["service"] }, { min_lines = 20 }] + +[testing.coverage] # reserved — parsed but not enforced yet +line = 80 + +[severity] # per-rule severity overrides +"style.function.max_lines" = "warning" + +[[override]] # scoped relaxation (e.g. legacy code) +paths = ["legacy/**"] +[override.style.function] +max_lines = 120 +``` + +## Rules + +| Rule id | Category | Checks | Default severity | +|---|---|---|---| +| `style.function.max_lines` | Style | Function length in lines | `warning` | +| `style.function.max_parameters` | Style | Parameter count from the signature (`self` excluded) | `warning` | +| `style.function.max_nesting` | Style | Nesting depth heuristic from flow markers | `warning` | +| `style.naming` | Style | Name matches the configured glob per kind | `warning` | +| `architecture.boundary` | Architecture | Resolved call graph edges across denied layer boundaries | `blocking` | +| `testing.missing_test` | Testing | Business logic with no reference from test paths | `required` | + +Severity model (least → most serious): `info`, `warning`, `required`, +`blocking`. Layers are mapped from file-path globs; boundary edges are +evaluated against the resolved call graph (caller file layer → callee file +layer). "Logic" for the testing rule is anything matching a `logic_selectors` +entry (a layer, or a minimum function size). + +## Scope resolution + +Policies resolve per file, following file → directory → repository order: +`[[override]]` blocks whose `paths` globs match a file are shallow-merged over +the base policy (scalars replace; rules/layers are appended). Use this to +relax rules in legacy areas without weakening them repo-wide. + +## CLI reference + +``` +codesmell check [paths...] [--diff ] [--format human|json] [--fail-on warning|required|blocking] +codesmell guide [path] +codesmell init +codesmell policy +``` + +- `--format json` emits the full report (`violations[]` with + `rule/severity/file/line/symbol/message/fix_hint`, plus a `summary`), for + machine and LLM consumption. +- `--fail-on` sets the severity threshold that makes the process exit `1` + (default: `required`, i.e. missing tests and boundary violations fail; + warnings alone do not). +- `--diff` accepts a unified diff (file path or `-` for stdin) and evaluates + only symbols overlapping the diff hunks — cheap change-aware validation. + +Human output is rustc-style: + +``` +error[architecture.boundary]: `place_order` (controller) calls `save_order` (repository): edge `controller -> repository` is denied + --> src/controllers/order_controller.rs:6 + hint: route `place_order` through an allowed layer instead of calling `repository` directly +``` + +## Agent workflow + +```text +codesmell guide (read conventions) + → LLM writes code + → codesmell check --diff - + → fix violations (sorted blocking → info, each with a fix_hint) + → re-check until clean +``` + +## Relationship with CodeGraph + +CodeGraph understands the software (AST, symbols, call graph, flows); +CodeSmell evaluates those facts against the team's engineering conventions. +CodeSmell builds a fresh in-memory CodeGraph per run — the persistent +`.codegraph` index is not required. + +## Not implemented yet (fast-follow) + +- Convention discovery (statistics → candidate policies with confidence) +- Coverage threshold enforcement +- Policy history / evolution +- Runtime/engineering policies (timeouts, retries via effect classification) diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 000000000..bf6e2040d --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,204 @@ +# Configuration reference + +Everything CodeGraph reads at runtime lives in `.codegraph/` next to your +project. This page documents `config.toml`, the ignore files, file filtering, +and the CLI flags that shape behavior. + +- `.codegraph/` layout: + +``` +.codegraph/ + db.sqlite SQLite backend (WAL, single file); db.lmdb/ directory when LMDB is selected + config.toml Languages, effect rules, storage backend, sandbox, embedding + .gitignore Pre-filled with "*" — the index is never committed + version CodeGraph version that created the directory +``` + +- `codegraph init` (CLI) / `codegraph_init` (MCP) creates the directory + idempotently and only writes `config.toml` if absent — your edits are + preserved on re-init. +- A missing or invalid `config.toml` silently falls back to defaults + (sqlite backend, auto headers, built-in effect rules). + +## config.toml + +### `[languages]` + +```toml +[languages] +headers = "auto" # "auto" (default) | "c" | "cpp" — how `.h` files are routed +``` + +`.h` resolution when both C and C++ grammars are compiled: + +| Mode | Behavior | +|---|---| +| `"auto"` (default) | Project hint first (C-only tree → C, C++-only tree → C++), then content sniffing of the first 8 KiB (`namespace`, `class`, `template`, `std::`, `public:`, …) | +| `"c"` | All `.h` parsed as C | +| `"cpp"` | All `.h` parsed as C++ | + +After changing, re-index (`codegraph init` or `codegraph_index`). + +### `[[effect_rules]]` + +Classify what a call does (`EffectType`) from its name. Config rules are +evaluated **before** the built-in defaults and first-match-wins, so they +override the table below. + +```toml +[[effect_rules]] +call = { prefix = "db." } # or { contains = "..." } or { exact = "..." } +effect = "sql_query" # one of the EffectType values below +``` + +- Matchers: `prefix` (call name starts with), `contains` (substring anywhere), + `exact` (case-sensitive full match). +- `effect` values (case-insensitive): `none`, `sql_query`, `sql_write`, + `cache_read`, `cache_write`, `http_call`, `event_emit`, `file_read`, + `file_write`, `log`. +- Rules with an unknown `effect` are skipped with a warning; the rest of the + config still loads. + +Built-in default rules (first match wins, top to bottom — the significant +groups; see `crates/codegraph-extract/src/languages/effects.rs` for the exact +ordered list): + +| Category | Matches (examples) | Effect | +|---|---|---| +| HTTP | `http.*`, `net/http.*`, `requests.*`, `RestTemplate`, `retrofit`, `WebClient`, `.Get(`, `.Post(`, `.Put(`, `.Patch(`, `.Do(`, `.NewRequest` | `http_call` | +| SQL read | `.Query`, `.QueryRow`, `.Raw`, `.Select`, `.Find`, `.First`, `.Model(` | `sql_query` | +| SQL write | `.Exec`, `.Insert`, `.Update`, `.Delete(`, `.Create(`, `.Save(`, `.Session` | `sql_write` | +| Cache | `.MGet`, `.HGet`, `.HGetAll`, `.Exists`, `.TTL` (read); `.MSet`, `.HSet`, `.Set`, `.Del(`, `.Expire` (write) | `cache_read` / `cache_write` | +| Events | `kafka.`, `rabbit`, `amqp`, `.Publish`, `.Send`, `.Produce`, `.Consume`, `.Subscribe`, `.Receive` | `event_emit` | +| Files | `os.`, `open(`, `.Open`, `.ReadFile`, `.ReadAll`, `FileInputStream`, `FileReader`, `BufferedReader` (read); `.WriteFile`, `.WriteString`, `.Create`, `.Mkdir`, `FileOutputStream`, `FileWriter` (write) | `file_read` / `file_write` | +| Logging | `log.*`, `slog.*`, `logging.`, `logger.`, `.Print*`, `.Info*`, `.Warn*`, `.Error*`, `.Debug*` | `log` | +| Fallback | `.Get` (without `(`) | `sql_query` | + +No match → `EffectType::None`. + +### `[storage]` + +```toml +[storage] +type = "sqlite" # sqlite (default) | lmdb | redis | memory | postgres | mysql +# dsn = "..." # verbatim override for any backend +# dsns = ["postgres://user:pass@db1:5432/codegraph", "postgres://user:pass@db2:5432/codegraph"] +# repo_id = 14028493579208694412 # generated by `codegraph init` — do not edit +``` + +| `type` | Aliases | DSN / notes | +|---|---|---| +| `sqlite` (default; unknown values also fall back here) | — | `sqlite:///.codegraph/db.sqlite` | +| `lmdb` | — | `lmdb:///.codegraph/db.lmdb` (directory) | +| `redis` | — | **requires** an explicit `dsn` (e.g. `redis://localhost:6379`) | +| `memory` | `in-memory`, `in_memory` | Ephemeral in-process index; nothing persisted; no file watcher | +| `postgres` | `postgresql`, `pg` | Multi-tenant, sharded — see below | +| `mysql` | `maria`, `mariadb` | Multi-tenant, sharded — see below | + +DSN resolution order: explicit `dsn` wins verbatim → otherwise derived from +`type` as above → `memory`/RDBMS without required fields → in-memory fallback +at open time. + +**Postgres / MySQL (multi-tenant, sharded)** — every table is partitioned by a +leading `repo_id` (random `u64` generated and written into `config.toml` by +`codegraph init`; self-healed on open if missing). Shard = `repo_id % len(dsns)` +with `dsns` = the shard list, or the single `dsn` when `dsns` is empty. The +schema is applied manually — run the files from `sql//` in order +(`001-initial-schema.sql`, `002-add-repos-registry.sql`) against every shard +before indexing. Build with `--features rdbms` (default for the `codegraph` +binary). See `sql/README.md` for the full design. + +### `[sandbox]` + +Defaults for the behavior sandbox (`codegraph_sandbox` / +`codegraph_diff_simulate` / `codegraph_origin_simulate`). See +[docs/sandbox.md](sandbox.md) for the full contract. + +```toml +[sandbox] +mock_dirs = ["sandbox/mocks"] # dirs (relative to root) with *.rhai mock files +loop_cap = 10 # max loop iterations per condition +branch_policy = "if_true" # "if_true" (default) | "if_false" +``` + +### `[embedding]` — semantic search (opt-in, off by default) + +```toml +[embedding] +# backend = "fastembed" # "hashing" | unset = disabled +# model = "bge-small-en-v1.5" # 384-dim, default +# cache_dir = "~/.cache/codegraph/embeddings" +# vss_extension = "~/.cache/codegraph/embeddings/vss" # SQLite-only sqlite-vss dir → HNSW ANN +# execution_provider = "cpu" # "cpu" (default) | "coreml" | "metal" (macOS) +``` + +- Unset `backend` = semantic search disabled; no model ever runs. +- With `fastembed`, `codegraph_search_symbol` gains `match: "semantic"` + (vector KNN) and `"hybrid"` (substring + semantic merged via Reciprocal + Rank Fusion). Vectors persist with the index. +- `vss_extension` points at a directory containing sqlite-vss `vector0`/`vss0` + extensions (SQLite backend only) — without it, KNN falls back to in-memory + brute force. Unset auto-probes `/vss`. +- `execution_provider` other than `"cpu"` requires building with + `--features fastembed,apple-accel` (macOS only). +- Pre-download the model for offline indexing: `codegraph embed --model + bge-small-en-v1.5` (compiled in with `--features fastembed`). +- If the model fails to load (no network / missing ONNX runtime) opening the + index errors out — there is no silent lexical fallback. + +## Ignore files and file filtering + +The walker (`crates/codegraph-extract/src/walker.rs`) uses `ignore::WalkBuilder`: + +- Hidden files/dirs skipped; `.gitignore` and `.git/info/exclude` honored + (including parent directories). +- `.codegraphignore` at the repo root adds project-specific patterns + (same syntax as `.gitignore`) — use it for generated code (`dist/`, + `*.min.js`, vendored SDKs). +- Only files with a known extension are parsed: + +| Language | Extensions | +|---|---| +| Rust | `rs` | +| Go | `go` | +| Python | `py`, `pyi` | +| Java | `java` | +| C# | `cs` | +| Ruby | `rb` | +| PHP | `php` | +| Scala | `scala`, `sc` | +| Swift | `swift` | +| Lua | `lua` | +| C | `c` | +| C++ | `cpp`, `cc`, `cxx`, `hpp`, `hh`, `hxx` | +| TypeScript | `ts`, `mts`, `cts` | +| TSX | `tsx` | +| JavaScript | `js`, `jsx`, `mjs`, `cjs` | +| Headers | `h` — routed per `[languages] headers` | + +- Files ≥ **4 MiB** or not valid UTF-8 are skipped (counted in index stats). + +## CLI flags that affect behavior + +| Flag | Effect | +|---|---| +| `--path ` | Workspace root override (default: cwd). For `serve`, only pre-seeds the stdio session root; HTTP sessions bind via `codegraph_init` | +| `codegraph init [--no-index]` | Create `.codegraph/`, generate `repo_id` for RDBMS backends, full re-index unless `--no-index` | +| `codegraph deinit` | Remove `.codegraph/` | +| `codegraph embed` | Pre-download the embedding model (needs `fastembed` feature) | +| `codegraph serve` flags | See [docs/mcp.md](mcp.md) — `--http`, `--addr`, `--allow-host`, `--allow-any-host`, `--api-key`, `--enable-observability`, `--format` | + +### File watcher + +Spawned by `serve` when the startup root is initialized **and** the backend has +a local DSN (sqlite/lmdb/redis) — `memory` and RDBMS backends get no watcher. +Debounce 500 ms (`notify-debouncer-full`), recursive; ignores changes under +`.codegraph/`, `.git/`, and paths matched by `.gitignore` (note: the watcher +does **not** consult `.codegraphignore`). Any relevant change triggers a +**full re-index** — there is no incremental mode. + +## Related + +- MCP server & tools: [docs/mcp.md](mcp.md) +- Sandbox & mocks: [docs/sandbox.md](sandbox.md) +- CodeSmell policy (`.codesmell/policy.toml`): [docs/codesmell.md](codesmell.md) diff --git a/docs/mcp.md b/docs/mcp.md new file mode 100644 index 000000000..b23bd8b7a --- /dev/null +++ b/docs/mcp.md @@ -0,0 +1,176 @@ +# MCP server reference + +CodeGraph exposes the semantic graph to AI agents over the Model Context +Protocol. One binary serves any MCP client — Claude Code, Cursor, Codex CLI, +opencode, Hermes, Antigravity — over **stdio** or **Streamable HTTP**. + +This page is the operator/dev reference: transports, client setup, the tool +catalog, and the token-saving conventions. The agent-facing usage guide that +ships **inside the binary** (returned as server `instructions` after +`initialize`) is [docs/codegraph.md](codegraph.md). + +## Running the server + +```sh +codegraph serve --mcp # stdio (1 process = 1 session) +codegraph serve --mcp --http --addr 0.0.0.0:8123 # Streamable HTTP (POST/GET/DELETE + SSE) +``` + +| Flag | Default | Notes | +|---|---|---| +| `--http` | off | Streamable HTTP transport, mounted at both `/` and `/mcp` | +| `--addr` | `0.0.0.0:8123` | Listen address | +| `--allow-host ` | — | Extra accepted `Host` headers (repeatable). Built-in allowlist: `localhost`, `127.0.0.1`, `::1` — this is rmcp's DNS-rebinding protection | +| `--allow-any-host` | off | Accept any `Host` header (trusted LAN/docker only, never public) | +| `--api-key ` | — | Accepted and parsed, but **not yet enforced** (auth is a known TODO; treat the HTTP port as unauthenticated) | +| `--enable-observability` | `true` | Accepted; `/health`, `/metrics` endpoints are a TODO and not mounted yet | +| `--format` | `minimize` | Default response encoding (`minimize` \| `medium`); overridable per session and per call | +| `--path ` | cwd | Pre-seeds the stdio session root. When the client starts the server from `/` (Claude Desktop) the session starts empty — the agent binds with `codegraph_init` | + +Notes: + +- **stdio**: one process = one session slot. Startup `--path` is only a + pre-seed; an empty session is fine — the agent binds the workspace with + `codegraph_init {"path": ...}`. +- **HTTP**: each connection (`mcp-session-id`) gets its own fresh + `CodegraphServer` via the factory — nothing is shared between connections + but the process. The agent binds the workspace root with `codegraph_init` + inside that connection. Legacy session mode is enabled: clients negotiating + a pre-2026-07-28 protocol still get sessions; SEP-2567 requests run + stateless. +- `tools/list` responses carry `ttlMs: 0` + `cacheScope: public` per + SEP-2549 / protocol 2026-07-28. + +## Client configuration + +stdio (any MCP client): + +```json +{ + "mcpServers": { + "codegraph": { + "command": "codegraph", + "args": ["serve", "--mcp", "--path", "/abs/path/to/project"] + } + } +} +``` + +HTTP (remote / Docker): + +```json +{ "type": "http", "url": "http://:8123/mcp" } +``` + +Per-client quirks worth knowing: + +| Client | Config location | Note | +|---|---|---| +| Claude Code | `~/.claude/settings.json` → `mcpServers.codegraph` | Starts servers from `/` — rely on `codegraph_init`, or pass absolute `--path` | +| Cursor | `.cursor/mcp.json` | Wrong cwd otherwise — always inject an absolute `--path` | +| Codex | `~/.codex/config.toml` → `[mcp_servers.codegraph]` | TOML edits must preserve sibling tables | +| opencode | `opencode.jsonc` | Preserve comments when editing | + +## Tool catalog (27 tools) + +### Session / admin + +| Tool | What it does | +|---|---| +| `codegraph_init` | Bind session to a workspace root (idempotent, creates `.codegraph/`); `index` defaults `false` (non-blocking); optionally sets session `detail`/`format` defaults | +| `codegraph_deinit` | Release the session (root → null); `.codegraph/` stays on disk; query tools refuse while unbound | +| `codegraph_index` | Full re-index of the bound workspace | +| `codegraph_status` | Index health: symbol/chain/edge/file counts | +| `codegraph_query_usage_report` | Server telemetry (calls/errors, answer bytes, estimated source bytes read); `reset: true` clears | + +### Search + +| Tool | What it does | +|---|---| +| `codegraph_search_symbol` | Name search; `match`: `contains` (default) / `prefix` / `suffix` / `exact` / `semantic` (opt-in KNN) / `hybrid` (RRF merge); kind filter; pagination | +| `codegraph_symbol` | Lookup by `id` or exact `name`; duplicates → `ambiguous` + match list, retry with `id` | +| `codegraph_search_by_annotation` | Symbols by annotation substring (e.g. `@RestController`), optional kind filter | +| `codegraph_search_by_call` | Functions calling a class/method name in their bodies — includes unresolved external calls, with per-call-site context | +| `codegraph_references` | Functions calling a library call whose name contains `query` | +| `codegraph_search_flow` | Functions whose call chain contains a pattern (comma tokens: marker names, symbol names, or numeric ids) | +| `codegraph_files` | Indexed files under a path prefix | +| `codegraph_dependencies` | Internal vs external module-prefix dependencies, sorted by call-site count | + +### Graph queries + +| Tool | What it does | +|---|---| +| `codegraph_callers` | Transitive callers (`depth`, default 1) | +| `codegraph_callees` | Direct callees | +| `codegraph_impact` | Transitive impact radius (`max_depth`, default 3) | +| `codegraph_flow` | Call chain: markers + callee names + call sites (line / condition / effect / args) | +| `codegraph_class_methods` | Methods of a class/interface/enum | +| `codegraph_class` | Class details with fields and methods as separate lists | +| `codegraph_list_classes` | All class symbols (paginated) | +| `codegraph_list_interfaces` | All interface symbols (paginated) | +| `codegraph_function_scope` | A function's parameters and local variables | +| `codegraph_context` | Composed context: search + callers + callees + optional source (markdown) | + +### Diff / sandbox + +| Tool | What it does | +|---|---| +| `codegraph_diff` | Unified diff → read-only DRAFT graph-impact report | +| `codegraph_sandbox` | Compile an entry function + in-flow callees to machine code (Cranelift JIT), run with Rhai mocks | +| `codegraph_diff_simulate` | Diff → sandbox run on current index vs temp index from `base_ref` (`git archive`); compare traces | +| `codegraph_origin_simulate` | Ref (default `HEAD`) vs working-tree sandbox run, no diff needed | + +See [docs/sandbox.md](sandbox.md) for the sandbox contract. + +## Common arguments + +- `detail`: `minimal` | `medium` (default) | `verbose` — per-call override of + the session default set at `codegraph_init`. +- `format`: `minimize` (default) | `medium` — per-call override of session / + startup default. +- `limit` / `offset`: pagination (defaults: `limit` 20 for most, 10 for + `references`, 5 for `context`). +- `timeout_ms` (default 20000; `0` = no limit) + `resume`: broad searches + (`codegraph_search_symbol`, `codegraph_search_flow`, `codegraph_references`, + `codegraph_search_by_annotation`, `codegraph_search_by_call`, + `codegraph_list_classes`, `codegraph_list_interfaces`) error on timeout with + a `"resume": ""` id — retry the **exact same call** plus the id to + continue without re-scanning. Resume ids are short-lived, in-process, and + tied to the query args; re-index or restart invalidates them. + +## Response conventions (token-lean by design) + +Full contract with examples lives in [docs/codegraph.md](codegraph.md#response-formats-binance-style-minimal); +summary: + +- **`format=minimize`** (default): symbol items are fixed 14-element + **positional arrays** `[id, name, kind, scope, scope_id, type_ref, + type_name, file, line, end_line, signature, doc, annotations, language]`; + `detail` is ignored. +- **`format=medium`**: keyed objects; `detail` selects fields + (`minimal` = id/name/kind/file/line, `medium` adds `signature`, `verbose` + = full Symbol). +- **Omission rule** (both formats): object keys holding default values + (`null`, `false`, `""`, `[]`, `{}`, and `0` for the sentinels `scope_id` / + `type_ref` / `end_line`) are omitted — *absent means default*. Arrays never + drop positions; counts (`total`, `limit`, `offset`) always stay. +- **Paths are workspace-relative** in responses. +- **Disambiguation**: duplicate names return `ambiguous: true` + `matches`; + retry with `id` alone. + +## Architecture + +`crates/codegraph-mcp` implements `rmcp::handler::server::ServerHandler` on the +official Rust SDK (`rmcp` v3.1.x; features `transport-io`, and `http` adds +`transport-streamable-http-server` + `axum`). Tool definitions are a static +`ToolDef` list (`tools.rs::tool_defs`) — the single source of truth for +`tools/list`; dispatch goes through `run_tool` → session admin tools → +`GraphApi` for queries → `SharedGraphIndex` + sboxes for the sandbox trio. +Session state (root binding, index handle, detail/format defaults, resumable +search cursors) lives in `session.rs`. + +## Related + +- Agent usage guide (embedded in the binary): [docs/codegraph.md](codegraph.md) +- Configuration: [docs/configuration.md](configuration.md) +- Sandbox & mocks: [docs/sandbox.md](sandbox.md) diff --git a/docs/sandbox.md b/docs/sandbox.md new file mode 100644 index 000000000..963f198d9 --- /dev/null +++ b/docs/sandbox.md @@ -0,0 +1,130 @@ +# Sandbox & simulation (with mocking) + +The behavior sandbox answers **"what does this flow actually do?"** before you +touch code: it compiles an entry function plus its in-flow callees to native +machine code (Cranelift JIT) and runs them against **Rhai mocks**, returning +the observed execution trace — mock call order, branch decisions, loop +iterations. + +What the trace captures — and what it doesn't: the sandbox follows flow +**structure**. Branch decisions come from `branch_policy` (the guard *text* is +never evaluated), loops run up to `loop_cap`, and **numeric arithmetic on +values is not modeled**. The reliable signal is the call/branch **sequence**: +an MR that adds/removes a call, a branch, or switches a callee shows up as a +sequence delta; a pure arithmetic change does not. + +## Configuration — `.codegraph/config.toml` + +```toml +[sandbox] +mock_dirs = ["sandbox/mocks"] # dirs (relative to workspace root) with *.rhai mocks +loop_cap = 10 # max loop iterations per condition (termination guarantee) +branch_policy = "if_true" # "if_true" (default) | "if_false" — anything else is an error +``` + +- Missing `[sandbox]` / config file → these defaults. +- Per-call arguments (`branch_policy`, `loop_cap`, `mocks`) override the + config for that one run. +- `[[effect_rules]]` at the top level of the same file is shared with the + extractor (schema in [configuration.md](configuration.md)). + +## The mock contract (Rhai) + +A mock is a Rhai function named after the callee, taking **one array of i64** +and returning an i64 (abstract value): + +```rhai +// sandbox/mocks/order.rhai +fn get_stock(args) { 100 } +fn insert_order(args) { args[0] * 2 } // bodies can read args +fn send_email(args) { 0 } +``` + +- Every `*.rhai` file under each configured `mock_dirs` entry is loaded and + merged into one mock library. +- **Inline mocks** (the `mocks` tool argument) map callee name → source and + deterministically **replace** file mocks of the same name: + - body-only source (`"77"`, `"args[0] * 10"`) is auto-wrapped as + `fn (args) { }`; + - a full `fn (args) { … }` script is used as-is. +- **Link-time fail-fast**: before compiling, the sandbox verifies that every + callee the flow will dispatch to a mock has one configured (file or inline). + Any unconfigured callee aborts with + `link failed — no mock configured for callee(s): a, b` — supply exactly + those in `mocks` (or a `*.rhai` file) and call again. +- A mock that is missing only at *run* time records the miss and returns `0`; + misses surface as `missing_mocks` in the result. + +## Run semantics + +- `branch_policy`: `"if_true"` takes the then-branch of every `if`, + `"if_false"` the else-branch. Guard text is never read. +- Loops: a per-condition counter stays true while `n <= loop_cap`. +- Switch: the first case is taken once, then false. +- Output trace: ordered mock invocations (`callee`, args, result), condition + decisions (`if`/`loop`/`switch`, index, result), and a rendered + `sequence` (`"if:1"`, `"loop:0"`, `"call:"`, …). + +## MCP tools + +All three share `args` (i64 array of entry arguments), `mocks`, `branch_policy`, +`loop_cap`. + +### `codegraph_sandbox` + +Run one entry flow on the current index. + +- `node` (symbol id) or `name` (substring → first Function/Method match). +- Returns `entry`, `entry_id`, `group` (entry + every resolvable chain + callee), `args`, `return`, `mocks` (in order), `conds`, `missing_mocks`, + `sequence`. + +### `codegraph_diff` + +Not a sandbox tool, but the companion that scopes them: parses a unified diff +(MR / patch / `git diff`) and returns a read-only **draft** of the graph +impact — touched symbols, flows carrying call sites on changed lines, marker +windows, and transitive callers. Line numbers use the **new** (b-) side of +each hunk. See [codegraph.md](codegraph.md#diff-draft--codegraph_diff) for the +response shape. + +### `codegraph_diff_simulate` + +For the functions a diff touches, run the entry flow **twice** — on the +current index (post-MR) and on a temporary index rebuilt from `base_ref` +(default `HEAD`, materialized via `git archive` — the workspace must be a git +repo) — then compare the traces. + +- Args besides `diff`: `entry` (function name; default: first function + affected by the diff), `base_ref`. +- Response: `{draft, entry, base_ref, affected_functions, before, after, + delta: {sequence_added, sequence_removed}}`. A function missing from + `base_ref` reports `before` without `present`; a callee without a mock + reports `link_error`. +- Read-only; the temp tree is always removed. + +### `codegraph_origin_simulate` + +The standalone "before vs now" comparison, without a diff: run the sandbox on +an entry flow at a git `ref` (default `HEAD`, e.g. `origin/main`) and on the +current working tree, then compare. Use it to check whether local uncommitted +edits change a flow's behavior. + +- `entry` (required): function name — resolved **by name** in each index + (symbol ids differ between the ref tree and the working tree). +- Response: `{draft, entry, ref, origin, working_tree, delta}`. + +## Where it lives + +- Engine: `crates/codegraph-sboxes` — config (`src/config.rs`), Rhai mock + library (`src/rhai.rs`), JIT codegen (`src/codegen.rs`), runtime + trace + (`src/runtime.rs`, `src/trace.rs`). +- Example mocks: `crates/codegraph-sboxes/tests/mocks/*.rhai`; integration + tests in `crates/codegraph-sboxes/tests/{control_flow,end_to_end}.rs`. +- MCP dispatchers: `crates/codegraph-mcp/src/tools.rs` + (`dispatch_sandbox`, `dispatch_diff_simulate`, `dispatch_origin_simulate`). + +## Related + +- Agent-facing guide with worked examples: [docs/codegraph.md](codegraph.md) +- Full config reference: [docs/configuration.md](configuration.md) diff --git a/docs/specs/01-bootstrap.md b/docs/specs/01-bootstrap.md index d2ac11e4e..c4b806cc7 100644 --- a/docs/specs/01-bootstrap.md +++ b/docs/specs/01-bootstrap.md @@ -1,26 +1,52 @@ -# Spec 01 — Bootstrap workspace - -**État**: ✅ done - -## Objectif - -Workspace Cargo compilable avec les 9 crates squelettes. Aucun comportement, juste structure. - -## Livré - -- `/Cargo.toml`: workspace resolver=2, `[workspace.package]` (version, edition, license, repo), `[workspace.dependencies]` centralisées (serde, rusqlite, tree-sitter + 15 grammaires, clap, tokio, notify, ignore, rayon, dirs, jsonc-parser, toml_edit). -- Profils: - - `release`: `lto="fat"`, `codegen-units=1`, `strip="symbols"`, `panic="abort"`. - - `release-small`: hérite + `opt-level="z"`. -- `rust-toolchain.toml`: channel stable + rustfmt + clippy. -- `.gitignore`: `/target`, `.codegraph/`, IDE noise. -- 9 crates avec `Cargo.toml` + `src/lib.rs` (ou `main.rs` pour le binaire) commenté TODO. +# Spec 01 — Workspace bootstrap + +**Status**: ✅ done — describes the shipped workspace. + +## Goal + +A Cargo workspace holding every CodeGraph component, with shared dependency +versions and release profiles centralized. + +## Workspace layout (11 crates) + +``` +crates/ + codegraph-core/ Error + semgraph model (Symbol, SymbolKind, chains, markers, EffectType) + codegraph-extract/ tree-sitter extractors (14 langs, feature-gated) + walker + Orchestrator + codegraph-graph/ GraphIndex: registry + chain/name engines + pluggable storage + embeddings + codegraph-context/ Markdown context composition (symbol + callers + callees + source) + codegraph-api/ GraphApi — async query facade over SharedGraphIndex + codegraph-sboxes/ Behavior sandbox: Cranelift JIT + Rhai mock runtime + codegraph-mcp/ MCP server (rmcp SDK, stdio + Streamable HTTP), 27 tools + codegraph-bench/ Criterion benches (search, storage, pipeline) + CodSpeed + codegraph-installer/ Multi-agent client installer (Claude Code, Cursor, Codex, opencode) + codegraph/ CLI binary (init/deinit/embed/serve) + file watcher + codesmell/ Team-convention linter consuming CodeGraph facts (lib + CLI) +``` + +## Conventions + +- Root `Cargo.toml`: `resolver = 2`, `[workspace.package]` (version, edition + 2021, license, repository), `[workspace.dependencies]` centralizing serde, + tree-sitter + grammars, clap, tokio, notify, ignore, rayon, camino, axum, + rmcp, globset, etc. Crates reference them via `version.workspace = true` / + `{ workspace = true }`. +- Profiles: `release` = `lto="fat"`, `codegen-units=1`, `strip="symbols"`, + `panic="abort"`; `release-small` inherits with `opt-level="z"`. +- `rust-toolchain.toml`: stable + rustfmt + clippy. MSRV 1.80 workspace-wide; + `codegraph-graph` overrides to edition 2024 (needs ≥ 1.85). +- `.gitignore` keeps `/target` and `.codegraph/` out of VCS. ## Validation -`cargo check --workspace` finit sans erreur (~6s clean rebuild). +CI (`ci.yml`) runs `cargo clippy --workspace --all-targets -- -D warnings` +(with `RUSTFLAGS: -D warnings`), `cargo test --workspace` with coverage, and a +slim no-default-features build check. -## Notes +## Historical note -- Versions tree-sitter grammars: `swift=0.7`, `scala=0.26`, `lua=0.5`, `kotlin=0.3`, le reste `0.23`. Certaines crates communautaires lèveraient des conflits — surveiller à l'ajout d'une grammaire neuve. -- Crate principal `codegraph` (binaire) — `Cargo.toml` workspace dir `crates/codegraph`. Nom du paquet sur crates.io reste `codegraph`. +The original plan had 9 crates including `codegraph-db` and +`codegraph-resolve`; both were folded away — storage moved into +`codegraph-graph` behind a `Storage` trait (spec 03), and resolution became an +ingest phase of `GraphIndex` (spec 05). `codegraph-api`, `codegraph-sboxes` +and `codesmell` (spec 11) were added later. diff --git a/docs/specs/02-core-types.md b/docs/specs/02-core-types.md index 5f90602a6..20bbcfb5e 100644 --- a/docs/specs/02-core-types.md +++ b/docs/specs/02-core-types.md @@ -1,26 +1,62 @@ -# Spec 02 — Core types (NodeKind, EdgeKind, errors) +# Spec 02 — Core types (semgraph model) -**État**: ✅ done +**Status**: ✅ done — implemented in `crates/codegraph-core/src/semgraph.rs` +(wire-breaking replacement for the original `NodeKind`/`EdgeKind` design). -## Objectif +## Goal -Types partagés stable entre toutes les crates. Source unique pour les chaînes serialisées dans DB/MCP. +One stable type vocabulary shared by extraction, storage, and MCP — the single +source for everything serialized into the DB and over the wire. -## Choix +## Model -- `NodeKind` / `EdgeKind`: enums C-like `#[derive(Serialize, Deserialize)]` `#[serde(rename_all = "snake_case")]`. Méthode `as_str(self) -> &'static str` pour insertion DB sans alloc. -- `Node`: id `i64` (rowid SQLite), `kind`, `name`, `qualified_name: Option`, `file: Utf8PathBuf` (camino — pas de `OsString` partout), `start_line`, `end_line`, `signature`, `docstring`, `language`. -- `Edge`: `from`, `to`, `kind`, `file: Option`, `line: Option`. -- `Error`: `thiserror`, variantes `Io`, `Db`, `Parse`, `Invalid`, `NotInitialized`, `Other`. `Result = std::result::Result`. +Every symbol gets a **global id** (`SymbolId = u64`, monotonic, starts at +`SYMBOL_BASE = 100`; ids `1..100` are reserved control-flow markers). A +function's call chain is a `Vec` mixing markers and callee symbol ids — +edges are *derived* from chains, not stored as a separate relation. -## Mapping avec archive +### Markers (`marker_name` / `marker_id` round-trip) -NodeKind (22): file, module, class, struct, interface, trait, protocol, function, method, property, field, variable, constant, enum, enum_member, type_alias, namespace, parameter, import, export, route, component. +`LOOP=1`, `RECURSIVE_CALL=2`, `IF_TRUE=3`, `IF_FALSE=4`, `BRANCH_END=5`, +`RETURN=6`, `LOOP_BACK=7`, `SWITCH_CASE=8`, `SWITCH_END=9`, `BREAK=10`, +`CONTINUE=11`, `THROW=12`. -EdgeKind (12): contains, calls, imports, exports, extends, implements, references, type_of, returns, instantiates, overrides, decorates. +### `SymbolKind` (12 values, replaces the 22-value `NodeKind`) -Strings exacts identiques à `archive/src/types.ts` — agents prompts existants restent valides. +`Function`, `Method`, `Class`, `Interface`, `Enum`, `Variable`, `Constant`, +`Parameter`, `Field`, `Module`, `File`, `Config` — serde snake_case, with +`as_str()` / `parse()`. -## Hors scope +### `Symbol` -Pas de méthode `Node::new()` — construction directe par struct literal jusqu'à ce qu'un besoin émerge. +`{ id, name, kind, scope: ScopeLevel, scope_id, type_ref, type_name, file, +line, end_line, signature, doc, annotations, language }` — `scope_id` is the +containment link (method → class, param/local → function; `0` = global), +`line..=end_line` gives the body span (LOC = `end_line − line + 1`). + +### Calls & effects + +- `CallRecord` — unresolved call: `{ caller_id, call_name, position, + arg_exprs, line, condition, is_loop_body, effect, effect_desc, + target_class, target_method }` (structural resolution hints). +- `EdgeMeta` — resolved edge: `(caller_id, callee_id)` + `position`, + `condition` (guard text), `effect`, `is_loop_body`, `is_recursive`. +- `EffectType` (10 values): `none`, `sql_query`, `sql_write`, `cache_read`, + `cache_write`, `http_call`, `event_emit`, `file_read`, `file_write`, `log` — + classified from callee names by `[[effect_rules]]` (see + [configuration.md](../configuration.md)). +- `EffectCallPattern`: `Prefix` / `Contains` / `Exact` (untagged serde) — + shared schema for `config.toml`. + +### Query-result projections + +`FlowResult` / `FlowCall`, `ResolveResult` (ambiguous-name protocol), +`SearchFlowResult`, `CallSiteResult`, `MemberInfo`, `ClassInfo`, +`FunctionScope`, `DependenciesReport`, `DbStats`, `FileInfo`, +`SymbolMatch` (contains/prefix/suffix/exact). + +## Error + +`thiserror` enum in `error.rs`: `Io`, `Db`, `Parse`, `Search`, +`DepthExceedsLimit`, `Invalid`, `NotInitialized`, `MissingMocks` (sandbox +link failure), `Other`; `Result` alias exported crate-wide. diff --git a/docs/specs/03-db-layer.md b/docs/specs/03-db-layer.md index 3be99549c..f8df6e6fb 100644 --- a/docs/specs/03-db-layer.md +++ b/docs/specs/03-db-layer.md @@ -1,71 +1,72 @@ -# Spec 03 — DB layer +# Spec 03 — Storage layer -**État**: pending +**Status**: ✅ done — implemented in `crates/codegraph-graph/src/storage.rs` +(+ `storage/{sqlite,lmdb,redis,postgres,mysql}.rs`). The originally planned +`codegraph-db` crate was folded into `codegraph-graph` behind a trait, and +FTS5 was replaced by in-memory radix engines. -## Objectif +## Goal -Couche SQLite minimale et rapide. Crate `codegraph-db`. +Pluggable persistence for the semgraph: entities (symbols, chains, call +records, files, embeddings) + chain radix streams, with one file-local +default (SQLite) and optional network / multi-tenant backends. -## Stack - -- `rusqlite` features `bundled` + `backup`. Bundled = SQLite statique → zero dep système. -- Pas de pool — SQLite WAL gère écriture mono, lectures parallèles depuis autres connexions. Un `Connection` par thread suffisant; pour les batches d'extraction, une connexion writer + N readers via `parking_lot::Mutex`. -- Schema versionné via table `meta(key, value)`; clé `schema_version`. - -## API publique +## Design ```rust -pub struct Db { conn: Mutex } - -impl Db { - pub fn open(path: &Utf8Path) -> Result; // create + migrate - pub fn open_read_only(path: &Utf8Path) -> Result; - pub fn close(self) -> Result<()>; - pub fn schema_version(&self) -> u32; - - // Writes (transaction-scoped) - pub fn upsert_file(&self, f: &FileRow) -> Result; - pub fn insert_nodes(&self, nodes: &[Node]) -> Result>; - pub fn insert_edges(&self, edges: &[Edge]) -> Result<()>; - pub fn delete_file_cascade(&self, file_id: i64) -> Result<()>; - - // Reads - pub fn search_nodes(&self, q: &str, limit: u32) -> Result>; - pub fn node_by_id(&self, id: i64) -> Result>; - pub fn nodes_by_name(&self, name: &str) -> Result>; - pub fn callers_of(&self, id: i64) -> Result>; - pub fn callees_of(&self, id: i64) -> Result>; - pub fn files_under(&self, prefix: &str) -> Result>; - pub fn stats(&self) -> Result; -} +pub trait Storage: Send + Sync { /* async trait, default no-ops */ } +pub trait Tx: Send { /* atomic radix mutations */ } ``` -## Schema - -`schema.sql` (déjà ébauché): -- `meta(key, value)`: schema_version, last_index_ts, indexer_version. -- `files(id, path, language, sha256, size, mtime, indexed_at)` — path unique. -- `nodes(id, kind, name, qualified_name, file_id, start_line, end_line, signature, docstring, language)` — indices sur `name`, `qualified_name`, `file_id`, `kind`. -- `edges(id, from_id, to_id, kind, file_id, line)` — indices `(from_id, kind)` et `(to_id, kind)`. -- `nodes_fts` virtual FTS5 sur `name, qualified_name, signature, docstring`, `content='nodes' content_rowid='id'`, tokenizer `unicode61`. -- Triggers `nodes_ai`, `nodes_ad`, `nodes_au` pour sync FTS↔table. - -## Migrations - -`fn migrate(conn: &mut Connection)`: -1. Lit `meta.schema_version` (NULL = fresh). -2. Pour chaque version `/` (`001-initial-schema.sql`, `002-add-repos-registry.sql`) — +no migrations run from the binary. Full design in `sql/README.md`. ## Validation -- Tests unitaires `crates/codegraph-db/tests/`: open temp, insert 100 nodes/edges, search FTS, delete cascade. -- `cargo bench` (à voir) pour mesurer `insert_nodes(1000)` latence — référence pour optimiser batch size. +- `crates/codegraph-graph/tests/{sqlite,lmdb,rdbms,redis}.rs` — reopen + round-trips (ingest → reopen → same query results), sharded DSN selection, + repo_id partitioning. +- `crates/codegraph-bench` storage benches + `docs/benchmarks/storage-perf.md`. -## Pièges +## Deviations from the original spec -- FTS5 triggers doivent passer `delete-then-insert` sur UPDATE (pattern documenté SQLite). -- `bundled` ajoute ~1.5MB au binaire — accepté. -- WAL nécessite que le système supporte `mmap` shared; OK Linux/macOS/Windows. +- No FTS5: name search is an in-memory radix `Search` over lowercase + names, rebuilt on open/ingest — sub-millisecond and backend-independent. +- No incremental `sync`: `ingest` always resets and rebuilds (full re-index). +- rusqlite was replaced by sqlx so one driver stack serves SQLite + RDBMS. diff --git a/docs/specs/04-extraction.md b/docs/specs/04-extraction.md index d1e9551ab..442458d66 100644 --- a/docs/specs/04-extraction.md +++ b/docs/specs/04-extraction.md @@ -1,101 +1,77 @@ -# Spec 04 — Extraction (tree-sitter natif) +# Spec 04 — Extraction (tree-sitter, declarative LangSpec) -**État**: pending +**Status**: ✅ done — implemented in `crates/codegraph-extract` +(`orchestrator.rs`, `walker.rs`, `config.rs`, `languages/*`). -## Objectif +## Goal -Parser un workspace en parallèle, émettre Nodes + Edges + FileRow vers la DB. 15 langages tree-sitter natifs + 3 extractors texte (Svelte, Vue, Liquid). Delphi DFM reporté. +Parse a workspace in parallel and emit per-file `ParseResult`s +(symbols with local ids, chains, call records) for `GraphIndex::ingest`. ## Architecture ``` -ExtractionOrchestrator - ├── FileWalker (ignore crate, gitignore, .codegraphignore) - ├── LanguageRegistry (path → Box) - ├── ParsePool (rayon) - │ └── for each file: - │ extractor.extract(source) -> ExtractResult - └── DbBatcher (chunks de 500, une tx par chunk) +Orchestrator::with_registry() + ├── walker::walk(root, parsers, config) ignore-crate walk → FileMatch { path, parser } + ├── parse_files (rayon par_iter) thread-local EffectClassifier installed per job + │ └── parse_one: fs::read (< 4 MiB, UTF-8) → parser.parse_file + └── index_all(root, &mut GraphIndex) parse + ingest (full re-index) ``` -## Trait extractor - -```rust -pub trait Extractor: Send + Sync { - fn language(&self) -> &'static str; // "typescript" - fn extensions(&self) -> &'static [&'static str]; - fn ts_language(&self) -> tree_sitter::Language; - fn extract(&self, source: &str, file: &Utf8Path) -> Result; -} - -pub struct ExtractResult { - pub nodes: Vec, // no id yet - pub edges: Vec, // refer to NodeDraft by local index - pub imports: Vec, // resolved later by codegraph-resolve -} -``` - -`NodeDraft` = `Node` sans `id`, `EdgeDraft` = indices locaux dans le Vec de nodes; orchestrator résoud après insert. - -## Langages +`ParseResult { path, language, bytes, lines, symbols, chains, calls }` — all +ids are **local per file** (start at `SYMBOL_BASE`); `ingest` remaps them to +global ids. Chain position `0` is a placeholder for an unresolved callee. -Un module par langage dans `src/languages/`: -- typescript.rs (gère aussi tsx via grammaire séparée du même crate) -- javascript.rs (jsx) -- python.rs -- rust.rs -- go.rs -- java.rs -- c.rs / cpp.rs -- csharp.rs -- ruby.rs -- php.rs -- scala.rs -- swift.rs -- kotlin.rs -- lua.rs +## Language support — 14 languages, one declarative `LangSpec` -Chacun: -1. Parse source en arbre tree-sitter. -2. Walk avec `tree-sitter::Query` quand possible (queries S-expr déclaratives) sinon visit récursif. -3. Émet nodes pour: déclarations (fn/class/struct/etc.), imports, exports. -4. Émet edges: `contains` (parent → enfant), `calls` (sites d'appel), `extends`/`implements`. +`typescript` (ts/mts/cts + tsx), `javascript` (js/jsx/mjs/cjs), `python`, +`rust`, `go`, `java`, `c`, `cpp` (cpp/cc/cxx/hpp/hh/hxx + `.h` routing), +`csharp`, `ruby`, `php`, `scala`, `swift`, `lua`. Each is a feature flag +(`lang-*`, default `all-langs`) registering a parser in `registry()`. -Queries stockées en `include_str!("queries/typescript/symbols.scm")` — fichiers `.scm` versionnés avec le code. +A `LangSpec` (`languages/common.rs`) declares, per language: -## File walker +- `decls` — (node kind → SymbolKind) mapping for declarations +- `func_kinds` / `class_kinds` / `param_kinds` / `annotation_kinds` +- `calls` — `CallRule`s with name/target extraction hooks +- marker rules — `if_kinds`, `loop_kinds`, `switch_*`, `return/break/ + continue/throw/try/except/finally_kinds` -`ignore::WalkBuilder` avec: -- `.gitignore` honoré -- `.codegraphignore` custom (suffix layer) -- `hidden(true)` (skip `.git`, `.node_modules` etc — `ignore` les a déjà) -- `parents(true)` pour héritage gitignore amont -- Filtre extension via `LanguageRegistry::extension_set()` +`run_spec` walks the tree in two passes: a **symbol pass** (scope stack, +Method reclassification, in-file type resolution) and a **chain pass** +(markers + placeholder-0 call sites + `CallRecord`s). Annotations are +extracted for Java (`annotation`, `marker_annotation`) and C#/PHP/Swift +(`attribute`). -## Parallélisme +`.h` files route between C and C++ via `[languages] headers` +(auto/c/cpp — project hint then content sniffing, see +[configuration.md](../configuration.md)). -`rayon::ThreadPoolBuilder` configuré sur `num_cpus`. Chaque worker: -- Reçoit `(PathBuf, &dyn Extractor)`. -- Lit fichier (`fs::read_to_string` — taille limite 4MB sinon skip). -- Hash sha256 du contenu pour `files.sha256`. -- Parse + extract. -- Pousse `(FileRow, ExtractResult)` dans un crossbeam channel. +## Effects -Thread principal lit le channel, batch 500 → `Db::insert_*` en transaction. +`EffectClassifier` (thread-local, installed per rayon job) maps call names to +`EffectType` — config `[[effect_rules]]` first, then the built-in default +table. Schema and defaults: [configuration.md](../configuration.md). -## Modes +## File walking -- `index_all(root)` — purge + reindex tout. -- `sync(root)` — compare sha256 par fichier; reindex seulement les changés. +`ignore::WalkBuilder`: hidden files skipped; `.gitignore` + +`.git/info/exclude` (incl. parents) honored; `.codegraphignore` as a custom +ignore layer. Post-filter: known extensions only; files ≥ 4 MiB or non-UTF-8 +skipped (counted in `ExtractStats.skipped`). -## Tests +## Project config -- Fixtures `tests/fixtures/typescript/sample.ts` etc. -- Assert: count nodes/edges, présence symbole précis, contains edge parent. -- `pr19-improvements.test.ts` archive → ré-utiliser fixtures comme regression suite. +`ExtractConfig::load(root)` reads `.codegraph/config.toml` (missing/invalid → +defaults): header language, effect classifier, storage settings. +`init_project` creates `.codegraph/` idempotently (`.gitignore` = `*`, +`version`, `config.toml` only if absent). -## Pièges +## Deviations from the original spec -- Tree-sitter `Language` n'est pas `Sync` pour certaines versions; wrap dans `parking_lot::Mutex` par thread OU créer parser par fichier (cheap). -- Encoding non-UTF8: skip avec warn. -- Fichiers générés (`*.min.js`, `dist/`, `build/`): filtrer par défaut via `.codegraphignore` template. +- 14 languages (kotlin grammar incompatible with tree-sitter 0.25; Svelte/ + Vue/Liquid text extractors were not ported). +- No `.scm` query files — extraction is a declarative `LangSpec` + generic + walker, not per-language queries. +- No sha256/incremental sync — always full re-index (spec 09). +- Emits symbols + chains + CallRecords, not Node/Edge rows. diff --git a/docs/specs/05-resolution.md b/docs/specs/05-resolution.md index 4b65d6f66..44c3a699e 100644 --- a/docs/specs/05-resolution.md +++ b/docs/specs/05-resolution.md @@ -1,91 +1,56 @@ -# Spec 05 — Reference resolution + frameworks +# Spec 05 — Call resolution -**État**: pending +**Status**: ✅ done — implemented as the resolve phase of +`GraphIndex::ingest` (`crates/codegraph-graph/src/lib.rs`). The originally +planned `codegraph-resolve` crate (import resolver + 17 framework resolvers) +was not ported; resolution today is name-based over the global registry. -## Objectif +## Goal -Transformer imports textuels et patterns de framework en edges précis (`imports`, `references`, `route → handler`). +Turn each chain's placeholder-`0` positions (unresolved `CallRecord`s) into +real callee symbol ids, so that chains, edges, and the call-name index are +consistent after ingest. -## Pipeline +## Resolution order -``` -Db (post-extraction) - ↓ -ImportResolver - ↓ -NameMatcher - ↓ -FrameworkResolvers (express, laravel, rails, fastapi, django, flask, - spring, gin, axum, aspnet, vapor, react-router, - sveltekit, vue-nuxt, cargo-workspace, nestjs, drupal) - ↓ -new edges + new route nodes inserted -``` +For every `CallRecord`, ingest tries in order: -## ImportResolver +1. **Structural hint** — `target_class` / `target_method` (e.g. a Java class + literal or receiver type captured at parse time) narrows the candidate set. +2. **Exact name match** — the full call name (`fmt.Println`, + `orderRepository.saveOrder`) against the global name index. +3. **Short name** — the last segment of the call name (`Println`, + `saveOrder`). +4. **Best candidate scoring** — when several symbols share the name: + `override +5` · `has-chain +5` (the candidate itself has a chain, i.e. is a + function) · `same-file +3`. Highest score wins. -Input: `RawImport { from_file, module_spec, imported_names }`. +Unresolved calls keep their placeholder `0` but **remain queryable** through +the inverted call-name index (`callers_by_call_name` — the window to the +"outside world" of libraries), and `codegraph_references` / +`codegraph_search_by_call` surface them like resolved ones. -Étapes: -1. **Relative** (`./foo`, `../bar`): join + résolution extension (`.ts → .tsx → /index.ts`...). -2. **Alias** (tsconfig `paths`, jsconfig, vite alias, cargo workspace members, pyproject src layout): lus une fois via `path-aliases.rs` à l'init du resolver. -3. **Bare module** (`react`, `lodash`): pas résolu — emis comme edge `imports → external` (target = node fictif `external:react` ou skip selon flag). +## Derived state -Output: edges `imports(file_node → file_node or symbol_node)`. +After resolution, ingest builds: -## NameMatcher +- `chains_map` — func id → chain (remapped local → global ids) +- `edges: HashMap<(caller, callee), EdgeMeta>` — position, guard condition, + effect, `is_loop_body`, `is_recursive` +- `call_names` — lowercase call name (+ type-qualified aliases) → call sites -Pour les appels `calls` où la cible n'a été identifiée que par nom à l'extraction, résolution post-pass: -- Cherche `nodes` de kind `function|method|class` avec `name = target_name`. -- Si 1 candidat dans le même fichier ou un fichier importé: lien direct. -- Sinon: skip (évite faux positifs). +All of it is rebuilt from chains + records on every ingest and on reopen. -## Frameworks +## What is explicitly *not* done -Un module par framework. Trait commun: +- No import-graph resolution (relative/alias/bare-module), no tsconfig path + aliases, no cargo-workspace member mapping. +- No framework route resolvers (express/laravel/rails/spring/gin/… → route + nodes). Route/handler knowledge is left to the LLM reading `codegraph_flow` + / `codegraph_search_by_annotation` output. -```rust -pub trait FrameworkResolver: Send + Sync { - fn name(&self) -> &'static str; - fn detect(&self, root: &Utf8Path) -> bool; // package.json scan, Gemfile, etc. - fn resolve(&self, db: &Db) -> Result; -} - -pub struct FrameworkArtifacts { - pub route_nodes: Vec, - pub edges: Vec, -} -``` - -### Patterns critiques (référence archive) - -| Framework | Détection | Pattern | -|---|---|---| -| Express | `express` dans package.json | `app.get('/x', handler)` → route node + ref edge | -| Laravel | `composer.json/laravel` | `Route::get(...)`, controller@method | -| Rails | `Gemfile/rails` | `routes.rb` DSL | -| FastAPI | `pyproject/fastapi` | `@app.get('/x')` décorateur | -| Django | `manage.py` | `urls.py` `path()` | -| Flask | `flask` dep | `@app.route('/x')` | -| Spring | `pom.xml` / gradle | `@GetMapping` etc | -| Gin | go.mod gin-gonic | `r.GET("/x", handler)` | -| Axum | Cargo.toml axum | `Router::new().route("/x", get(h))` | -| ASP.NET | `.csproj` | `[HttpGet("/x")]` | -| Vapor | `Package.swift` vapor | `app.get("x", use: h)` | -| React Router | `react-router` | `} />` | -| SvelteKit | `svelte.config.js` | `src/routes/**/+page.svelte` | -| Vue/Nuxt | `nuxt.config` | `pages/**/*.vue` | -| Cargo workspace | `[workspace]` | members glob → cross-crate imports | -| NestJS | `@nestjs/core` | `@Controller('x')` + `@Get('y')` | -| Drupal | `*.info.yml` | hooks + services.yml | - -Chaque framework émet `route` node avec `qualified_name = METHOD path` (ex `"GET /users/:id"`), edge `references → handler symbol`. - -## Tests - -`tests/frameworks-integration.rs` (équivalent archive). Fixture par framework avec 2-3 routes attendues. - -## Pièges - -- Détection multi-framework: un projet peut avoir Vue + Express; tous les resolvers qui détectent run, pas de mutex exclusion. -- Réentrant: appel `sync` ne doit pas dupliquer routes — purge edges de kind `references` issues des resolvers avant ré-exécution. Marqueur `meta.source='framework:express'` sur l'edge. +Rationale: the semgraph call chains already answer the questions agents ask +("who calls this", "what does this flow do"); a framework-specific resolver +layer added maintenance cost without changing agent behavior. If route nodes +become a requirement, they should be added as a post-ingest pass over +annotations + call patterns rather than a separate crate. diff --git a/docs/specs/06-graph-context.md b/docs/specs/06-graph-context.md index c86e8c9cb..4cff29411 100644 --- a/docs/specs/06-graph-context.md +++ b/docs/specs/06-graph-context.md @@ -1,91 +1,57 @@ -# Spec 06 — Graph traversal + context builder +# Spec 06 — GraphIndex, GraphApi & context builder -**État**: pending +**Status**: ✅ done — implemented in `crates/codegraph-graph` +(`GraphIndex`, `SharedGraphIndex`, `diff.rs`), `crates/codegraph-api` +(`GraphApi`), `crates/codegraph-context`. -## Objectif +## GraphIndex -Requêtes graphe haut niveau pour MCP/CLI: callers, callees, impact radius. Builder qui compose tout en markdown/json pour l'agent. +The main index: an in-memory registry (source of truth) over pluggable +storage (spec 03), plus two search engines: -## Crate `codegraph-graph` +- **Chain engine** `Search` — radix over call chains; callers are found + by substring-searching `[callee_id]` across chains (KMP + optional bloom + filters), not by BFS over a stored edge table. +- **Name engine** `Search` — radix over lowercase symbol names for + contains/prefix/suffix/exact. -```rust -pub struct Traversal<'a> { db: &'a Db } +Public query surface (sync unless noted): -impl<'a> Traversal<'a> { - pub fn callers(&self, node: NodeId, depth: u32) -> Result>; - pub fn callees(&self, node: NodeId, depth: u32) -> Result>; - pub fn impact_radius(&self, node: NodeId, max_depth: u32) -> Result; - pub fn path(&self, from: NodeId, to: NodeId, max_depth: u32) -> Result>>; -} -``` +| Method | Notes | +|---|---| +| `symbol_by_id` / `resolve_by_name_or_id` | id-first; duplicate names → `ambiguous` + matches | +| `callers(id, depth)` (async) / `callees(id)` (async) | transitive BFS on the chain engine / direct chain read | +| `flow(id)` (async) | chain + rendered descriptions + call sites | +| `search_flow(pattern)` (async) | functions whose chain contains ids/markers | +| `callers_by_call_name(query, limit)` (async) | call-name index, includes unresolved calls | +| `function_scope(id)` | parameters + locals via `scope_index` | +| `members_of` / `list_methods_of_class` / `get_class_info` | class structure | +| `list_symbols_by_kind` / `search_by_annotation` (+ `_resumable` variants) | paginated, deadline-aware | +| `files()` / `dependencies_report()` / `stats()` | topology + health | +| `diff_assess(&ParsedDiff, root)` (async) | unified diff → read-only `DiffReport` (touched symbols, affected flows with marker windows, transitive callers) — the engine behind `codegraph_diff` | -- BFS avec `VecDeque<(NodeId, u32 depth)>`, set visited `HashSet`. -- Edge kind filter: callers/callees → `calls`; impact → `calls|references|imports|extends|implements`. -- Limite dure: 5000 visités, retourne `Truncated` flag. +`SharedGraphIndex` (Arc + RwLock) adds `ensure_fresh()`: probes the storage +version and rebuilds when the index was re-written (watcher / another +process). -`ImpactReport`: -```rust -pub struct ImpactReport { - pub root: Node, - pub direct: Vec, // depth 1 - pub transitive: Vec, // depth 2..=max - pub by_kind: HashMap, - pub truncated: bool, -} -``` +## GraphApi (`crates/codegraph-api`) -## Crate `codegraph-context` +Async facade over `Arc` — every method ensures freshness +then delegates. Adds pagination (`Pagination`), resumable searches with +server-side cursors (`SearchSessionStore`, `SearchCursor` + +`SearchCursorPhase`) and the timeout/resume protocol surfaced by the MCP +tools, plus `context_markdown` (delegating to codegraph-context). -Compose les briques pour répondre "give me context for X" — analogue à `codegraph_context` MCP tool. +## codegraph-context -```rust -pub enum Format { Markdown, Json } - -pub struct ContextRequest { - pub query: String, // symbol name OR free-text topic - pub depth: u32, - pub include_source: bool, - pub format: Format, -} - -pub fn build(db: &Db, req: &ContextRequest) -> Result; -``` - -Algorithme (port du `archive/src/context/`): -1. `search_nodes(query)` → top N candidates par FTS rank. -2. Pour chaque candidate: charge node, callers (d=1), callees (d=1), file siblings. -3. Si `include_source`: charge slice `start_line..=end_line` depuis disque (cache LRU sur fichier). -4. Sérialise selon Format. - -## Format markdown - -``` -## `getName` — function — src/foo.ts:42 - -```ts - -``` - -**Callers** (3): -- `processUser` — src/users.ts:118 (calls) -- ... - -**Callees** (2): -- `formatString` — src/utils.ts:5 (calls) -- ... -``` - -## Format json - -Structure tagged identique surface MCP `codegraph_context` archive — agents prompts existants compatibles. +Composes the "give me context for X" answer used by `codegraph_context`: +search candidates → for each, the symbol + direct callers + direct callees + +optional on-disk source slice → markdown serialization. Token-lean by design. ## Tests -- Fixture: 4 fichiers TS avec chaîne d'appels `A → B → C → D`. -- Assert: `callers(D, depth=3)` retourne A,B,C. -- Assert: `impact_radius(A, max=2).by_kind` count exact. - -## Pièges - -- Charger source à la demande → IO sur traversal large; cache file→string LRU 32 entrées suffit. -- Tronquage profondeur: documenter le flag dans la sortie markdown ("⚠ truncated at depth 5"). +`crates/codegraph-api/tests/api.rs` seeds a synthetic graph via +`GraphIndex::open(tempfile)` + hand-built `ParseResult`s (symbols, chains, +call records) and exercises the whole query surface including resume +round-trips with `TIMEOUT_EXPIRE_IMMEDIATELY`; `codegraph-graph` keeps +inline unit tests for engines and ingest invariants. diff --git a/docs/specs/07-mcp-server.md b/docs/specs/07-mcp-server.md index 86f94ebc4..fe76a0533 100644 --- a/docs/specs/07-mcp-server.md +++ b/docs/specs/07-mcp-server.md @@ -1,79 +1,72 @@ -# Spec 07 — MCP server (stdio JSON-RPC) +# Spec 07 — MCP server -**État**: pending +**Status**: ✅ done — implemented in `crates/codegraph-mcp` on the official +Rust SDK (`rmcp` v3.1.x). The hand-rolled JSON-RPC design was replaced by the +SDK once it matured; the 9-tool surface grew to 27. -## Objectif - -Serveur MCP minimaliste sur stdio. Pas de SDK Rust officiel mature → hand-roll JSON-RPC 2.0 + framing MCP. ~300 LOC. - -## Protocole - -- Transport: stdin/stdout. Framing JSON-RPC en LSP-style? Non — MCP utilise une ligne JSON par message (LSP-style headers seulement pour le mode HTTP). Pour stdio: une ligne `\n`-terminée par message. -- Méthodes obligatoires: - - `initialize` → renvoie `serverInfo`, `capabilities.tools`, `instructions` (le contenu de `server-instructions.md`). - - `initialized` (notification, no-op côté serveur). - - `tools/list` → array de tools. - - `tools/call` → invoque le tool. - - `ping` → `{}`. - - `shutdown` (optionnel selon agent). - -## Tools - -| Nom MCP | Handler | Args | -|---|---|---| -| `codegraph_search` | `db.search_nodes` | `{ query, limit?, kind? }` | -| `codegraph_node` | `db.node_by_id` ou by_name | `{ id?, name? }` | -| `codegraph_callers` | `traversal.callers` | `{ node, depth? }` | -| `codegraph_callees` | `traversal.callees` | `{ node, depth? }` | -| `codegraph_impact` | `traversal.impact_radius` | `{ node, max_depth? }` | -| `codegraph_context` | `context::build` | `{ query, depth?, include_source?, format? }` | -| `codegraph_explore` | `context::explore` | `{ paths[], depth? }` | -| `codegraph_files` | `db.files_under` | `{ path? }` | -| `codegraph_status` | `db.stats` | `{}` | - -Chaque tool a un JSON Schema `inputSchema` exposé dans `tools/list`. +Full operator reference: [docs/mcp.md](../mcp.md). Agent-facing guide embedded +in the binary: [docs/codegraph.md](../codegraph.md). ## Architecture -```rust -pub struct McpServer { - db: Arc, - traversal: Arc>, // ... ou re-create par call -} - -impl McpServer { - pub async fn run(self, stdin: impl AsyncBufRead, stdout: impl AsyncWrite) -> Result<()>; -} -``` - -Boucle: -1. `read_line` → parse `JsonRpcMessage` (request/notification). -2. Dispatch async via `tokio::spawn` (un task par call — concurrence). -3. Réponse écrite avec `Mutex` pour sérialisation des writes. - -## Server instructions - -`include_str!("server-instructions.md")` — contenu identique à `archive/src/mcp/server-instructions.ts`. Renvoyé dans `initialize.result.instructions`. - -À garder en sync avec `instructions-template` de l'installer (spec 08). - -## Erreurs - -JSON-RPC 2.0 standard: -- `-32700` parse error -- `-32600` invalid request -- `-32601` method not found -- `-32602` invalid params -- `-32603` internal error -- `-32000..-32099` server-defined (NotInitialized → `-32001`) +`CodegraphServer` implements `rmcp::handler::server::ServerHandler` and holds +a `Session` (root binding + index handle + detail/format defaults), usage +telemetry, and a `SearchSessionStore` for resumable search cursors. + +- `get_info()` advertises `enable_tools()` + instructions from + `SERVER_INSTRUCTIONS` — `include_str!("../../../docs/codegraph.md")`, so + the shipped guide and the repo doc are the same file. +- `list_tools()` returns the static `ToolDef` list (`tools.rs::tool_defs`) — + single source of truth — with `ttl_ms(0)` + `cache_scope(Public)` + (SEP-2549 / protocol 2026-07-28). +- `call_tool()` rejects unknown names as `method_not_found`, then dispatches: + admin tools (`init`/`deinit`/`index`) on the session; queries through + `GraphApi`; the sandbox trio directly over `SharedGraphIndex` + sboxes. + +## Transports + +- **stdio** (`stdio.rs`) — `serve(rmcp::transport::io::stdio())`; one process + = one session. Startup `--path` is only a pre-seed (Claude Desktop starts + from `/` — the empty session binds via `codegraph_init`). +- **Streamable HTTP** (`http.rs`, feature `http`) — `StreamableHttpService` + on axum, mounted at `/` and `/mcp`; a factory builds a fresh + `CodegraphServer` per `mcp-session-id`; DNS-rebinding protection via + `with_allowed_hosts`; `with_legacy_session_mode(true)` keeps sessions for + pre-2026-07-28 clients. Known TODOs: bearer auth (`--api-key` parsed but + unenforced) and `/health`//`/metrics` endpoints (flag accepted, not + mounted). + +## Tools (27) + +Session/admin: `codegraph_init`, `codegraph_deinit`, `codegraph_index`, +`codegraph_status`, `codegraph_query_usage_report`. +Search: `codegraph_search_symbol` (contains/prefix/suffix/exact + opt-in +semantic/hybrid), `codegraph_symbol`, `codegraph_search_by_annotation`, +`codegraph_search_by_call`, `codegraph_references`, `codegraph_search_flow`, +`codegraph_files`, `codegraph_dependencies`. +Graph: `codegraph_callers`, `codegraph_callees`, `codegraph_impact`, +`codegraph_flow`, `codegraph_class_methods`, `codegraph_class`, +`codegraph_list_classes`, `codegraph_list_interfaces`, +`codegraph_function_scope`, `codegraph_context`. +Diff/sandbox: `codegraph_diff`, `codegraph_sandbox`, +`codegraph_diff_simulate`, `codegraph_origin_simulate` — full contract in +[docs/sandbox.md](../sandbox.md); engine in `codegraph-sboxes`. + +## Token-lean output conventions + +- `detail` minimal/medium/verbose (session default at `codegraph_init`, + per-call override) and `format` minimize/medium (startup → session → call). +- `minimize` (default): symbols as fixed 14-element positional arrays; + `medium`: keyed objects with `omit_defaults` (absent = default); + paths relativized to the workspace root. +- Broad searches take `timeout_ms` + `resume`: on timeout the tool errors + with a resume id; retry the same call + id to continue (short-lived, + in-process cursors in `SearchSessionStore`). ## Tests -- Integration: spawn `codegraph serve --mcp` sur fixture indexé, écris séquence `initialize` → `tools/call codegraph_search`, assert response. -- Pas de SDK client — fabrique requêtes JSON à la main. - -## Pièges - -- `tracing_subscriber` doit écrire sur **stderr** (jamais stdout — corrompt le protocole). -- Si DB pas init (`.codegraph/` absent): `initialize` OK mais tous tools renvoient `-32001 NotInitialized` avec message guidant `codegraph init`. -- Multi-instance: lockfile sur `.codegraph/db.sqlite` pour éviter writer concurrent. +Inline unit tests in `tools.rs` cover the formatting helpers (detail/format +parsing, positional-array schema index-by-index, `omit_defaults`, +relativization, tools/list cache fields); `http.rs` has an axum `oneshot` +smoke test asserting `initialize` returns `serverInfo.name = "codegraph"`. +Query behavior is tested one layer down in `crates/codegraph-api`. diff --git a/docs/specs/08-installer.md b/docs/specs/08-installer.md index 52d1bbd63..a8238d25d 100644 --- a/docs/specs/08-installer.md +++ b/docs/specs/08-installer.md @@ -1,76 +1,45 @@ # Spec 08 — Multi-agent installer -**État**: pending +**Status**: ✅ done — implemented in `crates/codegraph-installer` +(`src/targets/`, `instructions-template.md` embedded via `include_str!`). -## Objectif +## Goal -Configurer 5 agents (Claude Code, Cursor, Codex, opencode, Hermes) en une commande, idempotent, sans casser config existante. +Configure MCP clients (Claude Code, Cursor, Codex CLI, opencode) in one +command, idempotently, without breaking existing config. -## Trait +## Targets -```rust -pub trait AgentTarget: Send + Sync { - fn id(&self) -> &'static str; // "claude" - fn label(&self) -> &'static str; // "Claude Code" - fn detect(&self) -> DetectStatus; // NotInstalled | Installed | PartiallyInstalled - fn install(&self, opts: &InstallOpts) -> Result; - fn uninstall(&self) -> Result; -} - -pub enum DetectStatus { NotFound, Found, AlreadyConfigured } -pub enum InstallReport { Installed, Unchanged, Updated(Vec) } -``` - -## Cibles - -| Agent | Config | Notes | +| Target | Config written | Notes | |---|---|---| -| Claude Code | `~/.claude/settings.json` (global) ou `.claude/settings.local.json` (project) + `CLAUDE.md` | JSON, `mcpServers.codegraph` | -| Cursor | `.cursor/mcp.json` + `.cursor/rules/codegraph.mdc` | **Quirk**: cwd faux → injecter `--path` (absolu si project, `${workspaceFolder}` si global) | -| Codex | `~/.codex/config.toml` + `~/.codex/AGENTS.md` | TOML, table `[mcp_servers.codegraph]` — sérializer maison qui préserve siblings | -| opencode | `opencode.jsonc` ou `.json` + `~/.config/opencode/AGENTS.md` | Préfère `.jsonc`, edits via `jsonc-parser` pour préserver commentaires | -| Hermes | `~/.hermes/...` (TBD à partir d'archive `targets/hermes.ts`) | À documenter en porting | - -Chaque cible vit dans `crates/codegraph-installer/src/targets/{id}.rs`. - -## Shared - -- `instructions-template.rs`: une seule chaîne agent-agnostique (titre + tableau tools + chains). Source de vérité partagée avec `codegraph-mcp/server-instructions.md` — un test compare les deux contenus. -- `config_writer.rs`: helpers pour JSON / JSONC / TOML surgical edits. -- `toml.rs`: sérializer minimal pour `[mcp_servers.X]` qui préserve tables sœurs (cf. archive `targets/toml.ts`). - -## Détection installation existante - -`detect()`: -- Lit le fichier config s'il existe. -- Parse, check présence de la clé `codegraph` dans le bloc MCP. -- Retourne `Found` si présente et args valides, `NotFound` sinon, `Installed` si match exact attendu. - -## Idempotence +| Claude Code | `~/.claude/settings.json` → `mcpServers.codegraph` (+ agent instructions) | JSON surgical edit | +| Cursor | `.cursor/mcp.json` | **Quirk**: wrong cwd — always inject an absolute `--path` | +| Codex | `~/.codex/config.toml` → `[mcp_servers.codegraph]` | TOML edits preserve sibling tables | +| opencode | `opencode.jsonc` / `.json` | Comment-preserving edits via `jsonc-parser` | -Test obligatoire (spec depuis archive `__tests__/installer-targets.test.ts`): -- `install` deux fois → second call retourne `Unchanged`, fichier byte-equal après le premier. -- `uninstall` après `install` restaure fichier à l'état initial (avec une tolérance EOL). -- Tables/clés sœurs (`[mcp_servers.other]`, `mcpServers.other`) intactes. +Each target lives in `src/targets/{id}.rs` behind a common trait +(id/label/detect/install/uninstall). -## CLI +## Agent instructions -`codegraph install` (interactif via `dialoguer` ou `inquire`): -1. Détecte agents présents. -2. Multi-select prompt — coche par défaut ceux détectés. -3. Per-agent confirm + install. -4. Résumé final. +`INSTRUCTIONS_MD` (`src/instructions-template.md`) is the agent-agnostic +instruction block written next to the client config. The richer, always +current guide is [docs/codegraph.md](../codegraph.md) — also the file +embedded into the MCP server itself, so installer text and server +instructions no longer drift apart. -Flags: `--all` pour install non-interactif sur agents détectés. +## Idempotence contract -## Tests +- Installing twice → second run reports unchanged; file byte-equal after the + first. +- Sibling entries (`mcpServers.other`, `[mcp_servers.other]`) survive + untouched; comments in `.jsonc` survive. +- Uninstall removes only the `codegraph` entry. -- Per-target: parameterized contract suite. Pour chacun: fresh install, re-install (byte-equal), sibling preservation, uninstall reversal, partial-state recovery. -- ~50 tests cible. +Validated in `crates/codegraph-installer/tests/install.rs`. -## Pièges +## Entry point -- Cursor MCP working-dir: oublier `--path` casse silencieusement. -- Codex `~/.codex/config.toml`: arrays `[[mcp_servers]]` (table arrays) à preserver — pas le format qu'on écrit, mais on doit le rendre verbatim. -- opencode `.jsonc` peut contenir des commentaires importants — toujours passer par `jsonc-parser` edits. -- Permissions Windows sur `~/.claude/` — créer le dossier si absent. +Exposed as the `install` subcommand of the `codegraph` binary (interactive +multi-select of detected agents; `--all` for non-interactive). The original +"Hermes" target was dropped when its config format was never stabilized. diff --git a/docs/specs/09-cli-watcher.md b/docs/specs/09-cli-watcher.md index 14e6b2238..8b7cf08a3 100644 --- a/docs/specs/09-cli-watcher.md +++ b/docs/specs/09-cli-watcher.md @@ -1,67 +1,44 @@ # Spec 09 — CLI + file watcher -**État**: pending (squelette CLI fait) +**Status**: ✅ done — implemented in `crates/codegraph` +(`src/main.rs`, `src/watcher.rs`). -## Objectif +## Goal -Binaire `codegraph` final qui orchestre tout. Watcher live pour sync auto. +A deliberately minimal binary: workspace lifecycle + serving. All reading +and querying goes through MCP tools — the CLI intentionally has no +query/context/status subcommands. -## Sous-commandes +## Subcommands -| Cmd | Action | +| Command | What it does | |---|---| -| `codegraph` (no arg) | → `install` interactif | -| `install` | installer multi-agent (spec 08) | -| `init [-i/--index]` | crée `.codegraph/` + DB; `-i` lance indexation après | -| `uninit` | supprime `.codegraph/` après confirmation | -| `index` | full reindex | -| `sync` | incremental: rescan fichiers modifiés (compare mtime+sha256) | -| `status` | size DB, count nodes/edges/files, backend SQLite, dernière indexation | -| `query ` | search FTS, sortie tableau | -| `files [path]` | liste fichiers indexés sous path | -| `context ` | build markdown context, stdout | -| `affected ` | impact radius, stdout | -| `serve --mcp` | run MCP server stdio (spec 07) | -| `watch` | run file watcher en foreground (debug) | - -## Watcher - -- Crate `notify` + `notify-debouncer-full` (debounce ~500ms). -- Démarré automatiquement quand `serve --mcp` tourne — réindex live pendant que l'agent code. -- Filtre: même `ignore::WalkBuilder` qu'à l'index pour rejeter événements sur fichiers ignorés. -- Sur event: - - Create/Modify → enqueue `sync_file(path)`. - - Delete → `db.delete_file_cascade`. - - Rename → delete old + sync new. -- Worker tokio task dédié. - -## Output - -- `--json` global flag → toutes les commandes sortent JSON au lieu de texte humain. -- Couleurs via `anstream` (auto-detect TTY). -- Progress bar via `indicatif` pour `index` / `sync` long. - -## .codegraph layout - -``` -.codegraph/ - db.sqlite // schéma v1 - config.toml // ignore patterns custom, lang overrides - .gitignore // contient "*" (jamais commité) - version // texte: version du binaire ayant créé le dossier -``` - -## CLAUDE.md detection (existant archive) - -`init` détecte si project a CLAUDE.md / AGENTS.md / `.cursor/rules/` → propose `codegraph install` à la suite. - -## Tests - -- Smoke test: `init -i` sur fixture, `status` montre N nodes > 0, `query foo` répond. -- Watcher test: créer fichier dans tempdir, attendre debounce, assert node apparaît dans DB. - -## Pièges - -- `tracing` doit écrire stderr — `serve --mcp` corrompt le protocole sinon. -- Lockfile concurrent: `.codegraph/db.sqlite.lock` (advisory `fs2::FileExt::try_lock_exclusive`) pour bloquer `index` + `serve --mcp` simultanés sur le même writer. -- Signal handling: SIGINT pendant index → flush transaction en cours puis exit clean. +| `codegraph init [--no-index]` | Create `.codegraph/` (idempotent; preserves an existing `config.toml`), self-heal `repo_id` for RDBMS backends, full re-index unless `--no-index`. Live progress bar by default (`--no-progress` to disable) | +| `codegraph deinit` | Remove `.codegraph/` | +| `codegraph embed [--model ] [--cache-dir ]` | Pre-download the embedding model into the global cache (needs `fastembed` feature; default `bge-small-en-v1.5`) | +| `codegraph serve --mcp [--http …]` | Run the MCP server — stdio, or Streamable HTTP with `--addr` / `--allow-host` / `--allow-any-host` / `--api-key` / `--enable-observability` / `--format` (spec 07, [docs/mcp.md](../mcp.md)) | +| `codegraph install` | Multi-agent client installer (spec 08) | + +Global `--path ` overrides the workspace root (default cwd). + +## File watcher + +- Spawned by `serve` when the startup root is initialized **and** the + backend has a local DSN (sqlite/lmdb/redis). `memory` and Postgres/MySQL + get no watcher. +- `notify` + `notify-debouncer-full`, 500 ms debounce, recursive on the root. +- Ignores events under `.codegraph/`, `.git/`, and paths matched by the + repo's `.gitignore` (note: it does not consult `.codegraphignore`). +- Any relevant event triggers a **full re-index** (`GraphIndex::open(dsn)` → + `ingest`) — there is no incremental mode by design; simplicity over stale + state. `SharedGraphIndex::ensure_fresh()` picks the new version up on the + next query. + +## Deviations from the original spec + +- `sync`, `status`, `query`, `files`, `context`, `affected`, `watch` + subcommands were cut — the MCP tools cover all of them, and `status` lives + at `codegraph_status`. +- `init -i` became the inverse: indexing is the default, `--no-index` opts + out. +- No per-event create/modify/delete handling — debounce then full re-index. diff --git a/docs/specs/10-release.md b/docs/specs/10-release.md index ec750ebb9..33fa9d01e 100644 --- a/docs/specs/10-release.md +++ b/docs/specs/10-release.md @@ -1,77 +1,58 @@ -# Spec 10 — Release pipeline (GitHub Actions) +# Spec 10 — Release, packaging & CI -**État**: pending +**Status**: ✅ done (evolved) — CI in `.github/workflows/`, packaging in +`scripts/` + `packaging/aur`. -## Objectif +## CI (`ci.yml`) -Builds reproductibles cross-platform + GitHub Releases avec binaires attachés. `cargo install codegraph` fonctionne en parallèle. +Triggers: push to `main`, PRs, manual. `RUSTFLAGS: -D warnings`. -## Cibles +- **clippy** — `cargo clippy --workspace --all-targets -- -D warnings` + + a feature matrix run (`codegraph-graph` with postgres/mysql/redis). +- **test** — `cargo test --workspace --no-fail-fast` under coverage + instrumentation (`grcov` → lcov → Codecov), plus ignored storage + integration tests against real services (redis, postgres, mysql containers + with schemas from `sql/`). +- **slim** — verifies `cargo build -p codegraph --no-default-features` + compiles without the `rdbms` wiring. -| OS | Target triple | Runner | -|---|---|---| -| Linux x86_64 | `x86_64-unknown-linux-gnu` | ubuntu-latest | -| Linux x86_64 musl | `x86_64-unknown-linux-musl` | ubuntu-latest (cross) | -| Linux aarch64 | `aarch64-unknown-linux-gnu` | ubuntu-latest (cross) | -| macOS x86_64 | `x86_64-apple-darwin` | macos-13 | -| macOS aarch64 | `aarch64-apple-darwin` | macos-latest | -| Windows x86_64 | `x86_64-pc-windows-msvc` | windows-latest | +The workspace's newer crates (`codesmell`) are covered automatically as +workspace members. -Linux musl = bin statique zéro dep glibc → recommandé pour `curl | sh` install. +## Distribution -## Workflow `.github/workflows/release.yml` +- GitHub Releases with per-platform archives (Linux x86_64 musl + aarch64, + macOS x86_64 + arm64, Windows x86_64) — see the README install matrix. +- `scripts/install.sh` (curl | sh → `~/.local/bin`, override with + `CODEGRAPH_INSTALL_DIR`) and `scripts/install.ps1` (Windows). +- Arch Linux AUR package (`packaging/aur`, `codegraph-rs-bin`). +- From source: `cargo build --release -p codegraph` or + `cargo install --git … codegraph`. -Déclencheur: `push: tags: ['v*']`. +## Binary size & features -Steps: -1. Checkout. -2. `actions/cache` sur `~/.cargo`, `target/`. -3. Setup rust stable + target triple. -4. `cargo build --release --target $TRIPLE -p codegraph`. -5. Strip + UPX (optionnel — UPX casse macOS signing, à valider). -6. Archive: `tar.gz` Linux/macOS, `zip` Windows. -7. Checksums SHA256 par archive. -8. `gh release create $TAG --notes-file CHANGELOG_EXTRACT.md` (extract section `## [X.Y.Z]`). -9. `gh release upload` toutes les archives + `.sha256`. +Reality vs the original "<15 MB" target: ~**58 MB** stripped with **every** +backend bundled (SQLite, LMDB, Redis, Postgres/MySQL drivers, ONNX embedding +runtime) — accepted in exchange for a single zero-setup binary. The +`release-small` profile (`opt-level="z"`) remains for constrained targets. -## CI hors release `.github/workflows/ci.yml` +Feature flags (see README "Development" for the full list): `rdbms` (default +on the binary), `fastembed` (embed CLI + backend), `apple-accel` (macOS +CoreML), per-language `lang-*` on `codegraph-extract`. -- Push/PR: `cargo fmt --check`, `cargo clippy --workspace --all-targets -- -D warnings`, `cargo test --workspace`. -- Matrix: Linux + macOS + Windows. -- Bench (optionnel): `cargo bench` sur Linux, comparaison vs baseline stockée. +## crates.io (when publishing) -## Install script +Manual `cargo publish` in dependency order — the original order referenced +crates that no longer exist; the current one is: -`scripts/install.sh`: -```sh -#!/bin/sh -# detect OS+arch, download from GH Releases latest, verify sha256, install to ~/.local/bin -``` +1. `codegraph-core` +2. `codegraph-extract` · `codegraph-graph` (extract depends on graph) +3. `codegraph-context` · `codegraph-sboxes` · `codegraph-api` +4. `codegraph-mcp` · `codegraph-installer` +5. `codesmell` (depends on core/extract/graph) +6. `codegraph` (binary — users install this) -Equivalent `install.ps1` pour Windows. +## Deviations from the original spec -## crates.io - -`cargo publish` manuel (pas dans CI) pour éviter publish accidentel. Publier dans l'ordre des deps: -1. codegraph-core -2. codegraph-db -3. codegraph-extract, codegraph-resolve, codegraph-graph -4. codegraph-context -5. codegraph-mcp, codegraph-installer -6. codegraph (binaire — utilisateurs feront `cargo install codegraph`) - -## Tailles cibles - -- Bin Linux x86_64 stripped + LTO: viser **<15MB**. -- Si dépasse: profil `release-small` ou retirer langages exotiques (Lua/Scala/Swift via feature flags off). - -## Tests - -- Job `release-smoke`: après build, run `codegraph --version`, `codegraph init -i` sur fixture, assert exit=0. - -## Pièges - -- macOS notarization: hors scope MVP, signature ad-hoc OK. -- musl + rusqlite bundled: vérifier que `cc` est statique (devrait être OK avec bundled). -- Windows: `\r\n` dans archives — utiliser `7z` propre, pas `tar` GNU sur Win. -- CHANGELOG.md: réutiliser format archive (sections Added/Changed/Fixed) pour script d'extraction notes. +- musl is built via cross in CI; UPX was dropped (breaks macOS signing). +- No CHANGELOG auto-extraction job; release notes are written manually. diff --git a/docs/specs/11-codesmell.md b/docs/specs/11-codesmell.md new file mode 100644 index 000000000..3603add11 --- /dev/null +++ b/docs/specs/11-codesmell.md @@ -0,0 +1,64 @@ +# Spec 11 — CodeSmell (team-convention linter) + +**Status**: ✅ done (MVP) — implemented in `crates/codesmell` +(lib engine + CLI binary). User guide: [docs/codesmell.md](../codesmell.md). + +## Goal + +A policy engine that answers *"does this code look and behave like code this +team would write?"* — run like a linter. LLM agents read the conventions pack +(`codesmell guide`) before writing code and fix every reported violation +(`codesmell check`) afterwards; each violation carries a `fix_hint` so the +agent repairs instead of guessing. + +## Relationship to CodeGraph + +CodeSmell never re-implements code analysis. Each run parses the repository +fresh into a `GraphIndex::in_memory()` via `Orchestrator::parse_project` — +CodeGraph stays the understanding layer; only the persistent storage layer is +dropped (no `.codegraph` index required, no staleness). + +## Crate layout + +- `policy.rs` — `.codesmell/policy.toml` model (walked up from cwd), severity + `info | warning | required | blocking` with per-rule overrides, and + `[[override]]` blocks scoped by path globs (file → directory → repository + resolution). +- `index.rs` — `build_index(root)`: parse project → in-memory `GraphIndex`. +- `engine.rs` — `CheckScope` (`All` / `Paths` / `Diff`), candidate collection + (functions + methods, narrowed by scope), `evaluate` → severity-sorted + `CheckReport { violations, summary }`. +- `rules.rs` — the rule set (below). +- `guide.rs` — conventions-pack rendering + starter policy template + + AGENTS.md snippet. +- `main.rs` — CLI: `check [paths] [--diff ] [--format human|json] + [--fail-on …]`, `guide [path]`, `init`, `policy`. Human output is + rustc-style (`error[rule]: …` / ` --> file:line` / ` hint: …`); exit 1 + when violations ≥ the `--fail-on` threshold (default `required`). + +## Rules (MVP) + +| Rule id | Checks | Default severity | +|---|---|---| +| `style.function.max_lines` | Function LOC from `Symbol.line..=end_line` | warning | +| `style.function.max_parameters` | Parameter count parsed from the signature (depth-aware commas, `self` excluded) | warning | +| `style.function.max_nesting` | Nesting depth heuristic from chain markers | warning | +| `style.naming` | Name matches a glob per `SymbolKind`, optionally gated on `signature_contains` (e.g. async methods → `*Async`) | warning | +| `architecture.boundary` | Resolved call-graph edges across denied layer boundaries (layers = path globs) | blocking | +| `testing.missing_test` | Business logic (layer or min-size selectors) with no reference from `test_paths` files — via the call-name index | required | + +`--diff` scopes evaluation to symbols overlapping diff hunks +(`parse_unified_diff`, hunk new-line ranges) — change-aware validation. + +## Tests + +Unit tests for policy merge/severity/param-count/nesting; integration tests +over fixture repos (`tests/fixtures/rustshop` expecting every rule category, +`cleanshop` expecting zero violations, diff-scope narrowing). + +## Fast-follow (not built) + +Convention discovery (statistics → candidate policies with confidence), +coverage threshold enforcement, policy history/evolution, runtime/engineering +policies (timeouts/retries via effect classification), stored-index reuse as +a performance optimization. diff --git a/sonar-project.properties b/sonar-project.properties new file mode 100644 index 000000000..ade7e85e5 --- /dev/null +++ b/sonar-project.properties @@ -0,0 +1,14 @@ +sonar.projectKey=hungpham10_codegraph-rs +sonar.organization=hungpham10 + + +# This is the name and version displayed in the SonarCloud UI. +#sonar.projectName=codegraph-rs +#sonar.projectVersion=1.0 + + +# Path is relative to the sonar-project.properties file. Replace "\" by "/" on Windows. +#sonar.sources=. + +# Encoding of the source code. Default is default system encoding +#sonar.sourceEncoding=UTF-8