Pathweave

A structured knowledge extraction system for Python codebases, built on the Model Context Protocol (MCP). Pathweave parses source code into queryable entities, stores them as JSON or in PostgreSQL with pgvector embeddings, and exposes everything through MCP servers that AI agents can use for intelligent code analysis.

Overview

Pathweave transforms Python projects into structured, searchable knowledge through a multi-stage pipeline:

1. Parse — Tree-sitter AST analysis extracts files, functions, classes, methods, and TODO/FIXME issues
2. Store — Entities saved as JSON with search indices; Git commits and GitHub metadata integrated
3. Migrate (optional) — Entities migrated to PostgreSQL with 384-dimensional vector embeddings for semantic search
4. Serve — Two MCP servers expose tools to AI agents: one JSON-based (19 tools), one database-backed (6 tools with semantic search)

---

Quick Start

Installation

Requirements: Python 3.10+

pip install -r requirements.txt

Optional environment variables (.env file):

GITHUB_TOKEN=your_github_token        # For GitHub metadata extraction
MISTRAL_API_KEY=your_mistral_key      # For CrewAI agent tests

Basic Usage

Python API:

from src.generator import generate_resources

summary = generate_resources(
    project_path="path/to/project",
    output_dir="output",
    include_git=True,
    max_commits=50,
    include_github=True,
    github_repo="owner/repo"
)

print(summary['statistics'])
# {'total_entities': 383, 'files': 56, 'functions': 114, ...}

CLI:

python -m src.cli generate ./myproject -o output --git --github owner/repo

Full pipeline (generate + migrate to database):

python generate_and_migrate.py ./myproject --github owner/repo

Run Tests

python test/test_manual.py src         # Core pipeline validation (no API keys needed)
python test/test_crewai.py             # CrewAI agent test (requires MISTRAL_API_KEY)
python check_dependencies.py           # Verify all dependencies are installed

---

Architecture

Pipeline

Source Code → Parser (tree-sitter) → Entities (JSON) → Query Engine
                                          ↓
                                   PGVector Migration → PostgreSQL + Embeddings
                                          ↓
                                     MCP Servers → AI Agents (CrewAI)

Core Modules (src/)

Module	Purpose
parser.py	Tree-sitter AST parsing. Extracts files, functions, classes, methods, issues from Python source
storage.py	JSON persistence, Git commit extraction (GitPython), GitHub API integration
generator.py	Pipeline orchestration — runs parser → storage → indices → overview
generator_with_db.py	Extends generator with automatic PostgreSQL migration
query.py	In-memory query engine over JSON indices (search, filter, traverse)
cli.py	Click-based CLI interface for generation
migrate_to_pgvector.py	Migrates JSON entities to PostgreSQL with all-MiniLM-L6-v2 vector embeddings (384-dim)

Entity Types (src/entities/)

All entities inherit from BaseEntity with a unique ID format: type:filepath::name

Entity	Source	Key Data
FileEntity	Parser	imports, exports, line count
FunctionEntity	Parser	parameters, return type, docstring, complexity
ClassEntity	Parser	base classes, methods, attributes, decorators
MethodEntity	Parser	signature, parent class, decorators
IssueEntity	Parser	TODO/FIXME/HACK comments extracted from code
CommitEntity	Git	author, message, SHA, files changed, date
GitHubRepoEntity	GitHub API	stars, forks, language, topics
GitHubIssueEntity	GitHub API	title, state, labels, body
GitHubContributorEntity	GitHub API	commits, additions, deletions

MCP Servers (Python/)

Server	Tools	Backend	Use Case
MCP_Server_V2.py	19	JSON files	Full-featured: generation, search, commits, GitHub
MCP_Server_Database.py	6	PostgreSQL + pgvector	Semantic search, vector similarity, SQL queries
local_mcp_server.py	—	Legacy format	Original prototype (pre-v2)

MCP_Server_V2 tools include: generate_resources, find_entity, search_entities, get_entity_details, traverse_relations, multi_hop_query, get_statistics, search_functions, search_classes, get_class_methods, search_commits, get_commit_details, get_commit_statistics, list_commit_authors, get_github_repository, search_github_issues, get_github_issue_details, list_github_contributors, get_github_contributor_details

MCP_Server_Database tools: semantic_search, find_entity_by_id, search_by_name, get_relationships, find_commits_by_author, get_entity_statistics

---

Output Structure

After running generate_resources():

output/
├── overview.json              # Project statistics and metadata
├── entities/                  # Entity JSON files (MD5-hashed filenames)
│   ├── file/                  # Source file entities
│   ├── function/              # Standalone functions
│   ├── class/                 # Classes
│   ├── method/                # Class methods
│   ├── issue/                 # TODO/FIXME issues from comments
│   ├── commit/                # Git commits
│   ├── github_repo/           # Repository metadata
│   ├── github_issue/          # GitHub issues
│   └── github_contributor/    # Contributors
└── indices/                   # Search indices for fast lookups
    ├── functions_by_name.json
    ├── classes_by_name.json
    ├── files_by_path.json
    ├── commits_by_author.json
    ├── commits_by_sha.json
    ├── github_issues_by_number.json
    ├── github_contributors_by_login.json
    └── github_repo_info.json

---

PGVector Database (Optional)

Setup

# Start PostgreSQL with pgvector (Docker)
docker run --name vector_database \
  -e POSTGRES_PASSWORD=GeneralPass1987 \
  -e POSTGRES_DB=vectors \
  -p 5432:5432 -d pgvector/pgvector:pg16

# Install Python packages
pip install psycopg2-binary sentence-transformers torch

Migration

# Generate + migrate in one step
python generate_and_migrate.py ./myproject

# Or programmatically
from src.generator_with_db import generate_resources_with_db
generate_resources_with_db("./myproject", auto_migrate=True)

Database Schema

PostgreSQL "vectors"
├── entities                    # All entity data + embeddings
│   ├── entity_id (PK)         # e.g. "function:src/auth.py::validate"
│   ├── entity_type             # function, class, method, commit, etc.
│   ├── name, defined_in
│   ├── embedding vector(384)   # all-MiniLM-L6-v2 embeddings
│   ├── type_specific_data      # JSONB (signature, params, docstring...)
│   └── metadata                # JSONB
├── entity_relationships        # Relationship graph (calls, inherits, contains)
│   ├── source_entity_id (FK)
│   ├── target_entity_id (FK)
│   └── relationship_type
└── projects                    # Project metadata + statistics

Semantic Search Example

SELECT name, 1 - (embedding <=> query_vector) as similarity
FROM entities
WHERE entity_type = 'function'
ORDER BY embedding <=> query_vector
LIMIT 5;

Note: Only relationships between entities within the analyzed project are migrated. External library references (e.g., inheriting from BaseTool) are preserved in JSON but excluded from the database due to foreign key constraints.

---

Tests (test/)

Test	What it validates	Requirements
test_manual.py	Full pipeline: generate → query → search → statistics	None
test_crewai.py	CrewAI agent querying entities via @tool functions	MISTRAL_API_KEY
test_crewai_rich.py	CrewAI agent with 6 PGVector database tools (guided analysis of Rich library)	MISTRAL_API_KEY, PostgreSQL
test_crewai_rich_autonomous.py	Autonomous agent analysis — no guidance, open-ended task	MISTRAL_API_KEY, PostgreSQL
test_crewai_rich_baseline.py	Control experiment — same task with file-system tools instead of Pathweave	MISTRAL_API_KEY
test_vector_search.py	Standalone vector similarity search via pgvector	PostgreSQL
Crew_mcp_test.py	CrewAI + MCP server integration via stdio transport	MISTRAL_API_KEY

Benchmarks (benchmark/)

Benchmark	Purpose
benchmark_rich.py	Full pipeline benchmark on Textualize/rich: clone → parse → migrate → measure
benchmark_rich_part2.py	Supplementary: database statistics collection + semantic search benchmark

---

CrewAI Integration

from crewai import Agent, Task, Crew, LLM
from crewai.mcp import MCPServerStdio

agent = Agent(
    role="Code Analyst",
    goal="Analyze Python codebase",
    llm=LLM(model="mistral/mistral-large-latest", api_key=api_key),
    mcps=[MCPServerStdio(command="python", args=["Python/MCP_Server_V2.py"])]
)

task = Task(
    description="Find all functions that parse Python code and explain their relationships",
    agent=agent
)

crew = Crew(agents=[agent], tasks=[task])
result = crew.kickoff()

---

Project Structure

Pathweave/
├── src/                              # Core library
│   ├── __init__.py
│   ├── parser.py                     # Tree-sitter AST parser
│   ├── storage.py                    # JSON persistence + Git/GitHub integration
│   ├── generator.py                  # Pipeline orchestrator
│   ├── generator_with_db.py          # Generator + PostgreSQL migration
│   ├── query.py                      # Query engine (JSON indices)
│   ├── cli.py                        # Click CLI
│   ├── migrate_to_pgvector.py        # PGVector migration + embeddings
│   └── entities/                     # Entity dataclasses
│       ├── base.py                   # BaseEntity
│       ├── file.py, function.py, class_.py, method.py
│       ├── issue.py, commit.py
│       └── github_repo.py, github_issue.py, github_contributor.py
│
├── Python/                           # MCP servers
│   ├── MCP_Server_V2.py             # JSON-based server (19 tools)
│   ├── MCP_Server_Database.py       # PGVector server (6 tools, semantic search)
│   └── local_mcp_server.py          # Legacy server (prototype)
│
├── test/                             # Test suite
│   ├── test_manual.py               # Core pipeline validation
│   ├── test_crewai.py               # CrewAI + JSON tools
│   ├── test_crewai_rich.py          # CrewAI + PGVector tools (guided)
│   ├── test_crewai_rich_autonomous.py  # Autonomous agent experiment
│   ├── test_crewai_rich_baseline.py # Baseline comparison (no Pathweave)
│   ├── test_vector_search.py        # Semantic search demo
│   └── Crew_mcp_test.py             # MCP stdio integration
│
├── benchmark/                        # Performance benchmarks
│   ├── benchmark_rich.py            # Full pipeline benchmark (Rich library)
│   └── benchmark_rich_part2.py      # DB stats + search benchmark
│
├── output/                           # Generated resources (gitignored)
├── generate_and_migrate.py           # CLI: generate + migrate in one step
├── check_dependencies.py             # Dependency verification
├── requirements.txt
└── .env                              # API keys (gitignored)

---

Requirements

Core (required):

• tree-sitter + tree-sitter-python — AST parsing
• click — CLI interface

Optional integrations:

• GitPython — Local Git commit extraction
• requests + python-dotenv — GitHub API integration
• mcp — MCP server protocol
• crewai — AI agent framework

PGVector database (optional):

• psycopg2-binary — PostgreSQL adapter
• sentence-transformers — Vector embeddings (all-MiniLM-L6-v2)
• torch — Required by sentence-transformers

See requirements.txt for pinned versions.

---

License

MIT License