codetwine

A CLI tool that analyzes source code with tree-sitter, extracts key definitions and dependencies, generates design documents via LLM, and consolidates everything into a single unified JSON. The unified JSON can be used as input material for LLM-powered code search and Q&A, such as RLM, GraphRAG, and agent search.

Dependencies are extracted at the symbol level (functions, classes)
Design documents are generated per file, taking dependencies into account
Outputs dependency graphs in Mermaid format

🧬 Supported Languages

Python
Java
JavaScript
TypeScript
C
C++
Kotlin

🚀 Quick Start

Prerequisites

Python 3.11 or higher
uv package manager
API key for an LLM provider (Anthropic / OpenAI / Google, etc. Not required when using local LLMs like Ollama)

1. Installation

git clone https://github.com/yumeiriowl/codetwine.git
cd codetwine
uv sync

2. Configuration

cp .env.example .env

Set the following in the .env file:

# LLM API key (can be omitted for providers that don't require authentication, e.g. Ollama)
LLM_API_KEY=your-api-key-here

# LLM model name (specify the provider as a prefix in litellm format)
#   Anthropic:  anthropic/<model>
#   OpenAI:     openai/<model>
#   Google:     gemini/<model>
#   Ollama:     ollama/<model>
LLM_MODEL=anthropic/claude-sonnet-4-6

# Root directory of the project to analyze (absolute path)
DEFAULT_PROJECT_DIR=/path/to/your/project

# LLM output language (default: English)
OUTPUT_LANGUAGE=English

3. Usage

Basic Execution

uv run main.py

Analyzes the project set in DEFAULT_PROJECT_DIR in .env and outputs results to the output/ directory.

Specifying Project and Output Directories

uv run main.py --project-dir /path/to/your/project --output-dir /path/to/output

Argument	Description	Default
`--project-dir`	Root directory of the project to analyze	`DEFAULT_PROJECT_DIR` from `.env`
`--output-dir`	Output directory for analysis results	`DEFAULT_OUTPUT_DIR` from `.env` (defaults to `output/` if not set). When only `--project-dir` is specified, `DEFAULT_OUTPUT_DIR` is ignored and `output/` is used

⚙️ Configuration Options

The following options can be configured in the .env file.

LLM Settings

Variable	Description	Default
`LLM_API_KEY`	API key for the LLM provider	None
`LLM_MODEL`	Model name in litellm format (required when `ENABLE_LLM_DOC=True`)	None
`LLM_API_BASE`	API base URL (set when using non-standard endpoints, e.g. Ollama, Azure)	Not set
`OUTPUT_LANGUAGE`	Output language for design documents	`English`
`DOC_MAX_TOKENS`	Token limit for LLM output	`8192`

Path Settings

Variable	Description	Default
`DEFAULT_PROJECT_DIR`	Root directory of the project to analyze	Repository root
`DEFAULT_OUTPUT_DIR`	Output directory for analysis results	`output/`
`DOC_TEMPLATE_PATH`	Path to the design document template JSON file	`doc_template.json`

Performance Settings

Variable	Description	Default
`MAX_WORKERS`	Number of parallel workers for document generation	`4`
`MAX_RETRIES`	Number of retries for LLM API calls	`3`
`RETRY_WAIT`	Wait time in seconds between retries	`2`

Analysis Settings

Variable	Description	Default
`ENABLE_LLM_DOC`	Enable/disable LLM design document generation (`True` / `False`)	`True`
`SUMMARY_MAX_CHARS`	Maximum character count for summaries	`600`
`EXCLUDE_PATTERNS`	Patterns to exclude during file traversal (comma-separated, fnmatch format)	`__pycache__,.git,.github,.venv,node_modules`

🔄 High-Level Processing Flow

Build the project-wide dependency graph
- Collects source files with supported extensions from the target directory
- Analyzes import statements in each file and identifies inter-file dependencies
Detect changed files
- Compares source file hashes with the previous output to identify changed files for reprocessing
Extract dependency information for each file
- Generates a syntax tree with tree-sitter and extracts definitions (functions, classes, etc.)
- Based on the dependency graph built in step 1, extracts callee and caller file paths, line numbers, and source code
Generate design documents via LLM
- Sorts files in topological order, processing from files with no dependencies toward dependent files
- Passes each file's source code, dependency information, and callee document summaries to the LLM, generating a design document section by section according to the template (doc_template.json)
- Generates a summary for each file. This summary is included as input when generating documents for subsequent dependent files, giving the LLM knowledge of each dependency's role and behavior
Save all outputs as JSON
- Saved to <output directory>/<project name>
- All dependencies and design documents are consolidated into a single JSON (project_knowledge.json)
- Dependency graphs and design documents are also output as Markdown for readability

Note: LLM API calls are only made in step 4. No LLM is used in other steps.

📁 Output Files

Running the tool generates the following files in <output directory>/<project name>/ (default: output/<project name>/).

File	Description
`project_knowledge.json`	Consolidated JSON of all file dependencies and design documents
`project_dependency_summary.json`	Consolidated JSON of the dependency graph + per-file summaries
`dependency_graph.md`	Dependency graph in Mermaid format
`<filename>/file_dependencies.json`	Per-file definition and dependency information
`<filename>/doc.json`	Per-file design document (JSON format)
`<filename>/doc.md`	Per-file design document (Markdown format)
`<filename>/<original filename>`	Copy of the original source code

⚠️ Dependency Analysis Limitations

Dependency extraction is performed through static syntax analysis with tree-sitter, parsing import statements (including #include) in source code to identify inter-file dependencies. Dependencies may not be detected or may be incomplete in the following cases.

Common to All Languages

Dynamic imports: Patterns that construct module names as strings at runtime cannot be detected
- Python: importlib.import_module(name), __import__(name)
- JavaScript/TypeScript: import(variable)
- Java: Class.forName("com.example.Foo")

JavaScript / TypeScript

Build tool path aliases: Path aliases such as @/, ~/ defined in Webpack, Vite, tsconfig, etc. cannot be resolved. Imports using aliases are not recognized as project files and are missing from dependencies

Java / Kotlin

Wildcard imports: import com.example.* is detected as an import statement, but individual class files cannot be resolved, so they are not recognized as dependencies
Implicit same-package references: In Java/Kotlin, classes in the same package can be referenced without imports. Detection uses regex matching with the assumption that file names match class names, so cases with multiple classes in one file may be missed

C / C++

Build system include paths: Include paths added via CMake or Makefile -I options are not considered. Headers that cannot be resolved from the project root or current directory as relative paths are not detected as dependencies

♻️ Incremental Processing

On subsequent runs, only the changed files and their affected scope have their design documents regenerated.

Change detection: Compares SHA256 hashes of source files with the copies from the previous output to detect changed files
Dependency information: Re-extracted for all files every run to ensure consistency
Design documents: Only the changed files and files that depend on them (dependents) are regenerated; all others reuse previous results
Completeness check: Even for unchanged files, if the existing doc.json has missing sections or an empty summary (e.g. due to a previous LLM API failure), it is treated as incomplete and regenerated

📋 Output JSON Schema

project_knowledge.json

Consolidated JSON integrating all file dependencies and design documents.

{
  "project_name": "string",
  "project_dependencies": [
    {
      "file": "string",
      "summary": "string|null",
      "callers": ["string"],
      "callees": ["string"]
    }
  ],
  "files": [
    {
      "file": "string",
      "file_dependencies": {},
      "doc": {}
    }
  ]
}

Field	Type	Description
`project_name`	string	Project name
`project_dependencies[].file`	string	Path of the source file copied to the output directory
`project_dependencies[].summary`	string\|null	Summary of the file (null when design document is not generated)
`project_dependencies[].callers`	string[]	Paths of dependent files copied to the output directory
`project_dependencies[].callees`	string[]	Paths of dependency files copied to the output directory
`files[].file`	string	Path of the source file copied to the output directory
`files[].file_dependencies`	object	Same structure as file_dependencies.json (excluding `file` field)
`files[].doc`	object	Same structure as doc.json (excluding `file` field)

project_dependency_summary.json

Consolidated JSON of the dependency graph and per-file summaries.

{
  "project_name": "string",
  "files": [
    {
      "file": "string",
      "summary": "string|null",
      "callers": ["string"],
      "callees": ["string"]
    }
  ]
}

Field	Type	Description
`project_name`	string	Project name
`files[].file`	string	Path of the source file copied to the output directory
`files[].summary`	string\|null	Summary of the file
`files[].callers`	string[]	Paths of dependent files copied to the output directory
`files[].callees`	string[]	Paths of dependency files copied to the output directory

file_dependencies.json

Per-file definition and dependency information.

{
  "file": "string",
  "definitions": [
    {
      "name": "string",
      "type": "string",
      "start_line": 0,
      "end_line": 0,
      "context": "string"
    }
  ],
  "callee_usages": [
    {
      "name": "string",
      "from": "string",
      "target_context": "string",
      "lines": [0]
    }
  ],
  "caller_usages": [
    {
      "name": "string",
      "file": "string",
      "usage_context": "string",
      "lines": [0]
    }
  ]
}

Field	Type	Description
`file`	string	Path of the source file copied to the output directory
`definitions[].name`	string	Function/class name
`definitions[].type`	string	Definition type (tree-sitter node type, varies by language. Python: `function_definition`, `class_definition` / Java: `class_declaration`, `method_declaration` / JS/TS: `function_declaration`, `class_declaration`, etc.)
`definitions[].start_line`	int	Start line number
`definitions[].end_line`	int	End line number
`definitions[].context`	string	Full source code of the definition
`callee_usages[].name`	string	Name of the used symbol
`callee_usages[].from`	string	Path of the dependency file copied to the output directory
`callee_usages[].target_context`	string	Full source code of the dependency symbol
`callee_usages[].lines`	int[]	Line numbers of usage within this file
`caller_usages[].name`	string	Name of the symbol being used
`caller_usages[].file`	string	Path of the dependent file copied to the output directory
`caller_usages[].usage_context`	string	Source code of the usage location in the dependent
`caller_usages[].lines`	int[]	Line numbers of usage in the dependent file

doc.json

Per-file design document.

{
  "file": "string",
  "summary": "string",
  "sections": [
    {
      "id": "string",
      "title": "string",
      "content": "string"
    }
  ]
}

Field	Type	Description
`file`	string	Path of the source file copied to the output directory
`summary`	string	Summary of the file
`sections[].id`	string	Section identifier (corresponds to id in doc_template.json)
`sections[].title`	string	Section heading
`sections[].content`	string	Section body (Markdown format)

🎨 Customizing the Design Document Template

Edit doc_template.json to customize the section structure and LLM instructions for design documents.

{
  "sections": [
    {
      "id": "overview",
      "title": "Section heading",
      "prompt": "Instruction text for the LLM"
    }
  ],
  "summary_prompt": "Instruction for generating the overall summary"
}

Operation	Method
Add section	Add a new object to the `sections` array
Remove section	Remove the corresponding element from the `sections` array
Change instructions	Edit the text in the `prompt` field
Change summary instructions	Edit the `summary_prompt` field
Use a different template	Specify the path in `DOC_TEMPLATE_PATH` in `.env`

When you modify the template sections, existing design documents whose section structure no longer matches the template are automatically regenerated on the next run.

✏️ Manual Editing of Design Documents

You can manually edit the output doc.md and have it automatically reflected in doc.json on the next run.

Edit output/<project name>/<filename>/doc.md with a text editor
On the next uv run main.py execution, if doc.md has a newer timestamp than doc.json, the Markdown section content is parsed and applied to doc.json

Notes for editing:

Do not delete or rename ## Section heading lines. The parser uses them as section delimiters, and without headings, parsing will not work correctly
The body text below section headings can be freely edited

⏭️ Running Without Design Document Generation

To output only dependency information without generating LLM design documents, set ENABLE_LLM_DOC=False in .env.

ENABLE_LLM_DOC=False

The design document generation step is skipped. Dependency information (file_dependencies.json and source file copies) is still generated for each file, along with project_knowledge.json, project_dependency_summary.json, and dependency_graph.md. Since no LLM is used, it can run without API keys or model configuration.

💡 Usage Example: RLM QA Agent

examples/rlm_qa/ contains a sample that performs interactive Q&A against project_knowledge.json as a usage example for the consolidated JSON. It uses dspy's RLM and PythonInterpreter to generate answers by manipulating the JSON with Python code.

Sample Output

examples/sample_output/ contains sample output produced by analyzing the codetwine repository itself. This output was generated using a Python-specific template (examples/doc_template_python.json) created for this sample. rlm_qa_agent.py references this output by default, so you can try out RLM QA immediately without running any analysis.

Note: The file field paths in project_knowledge.json refer to sources copied into the output directory and differ from the original source tree paths (e.g. codetwine/import_to_path.py → codetwine/import_to_path_py/import_to_path.py).

Additional Prerequisites

Deno runtime
dspy package (install with uv sync --extra examples)

How to Run

uv run python examples/rlm_qa/rlm_qa_agent.py

By default, the sample output in examples/sample_output/ is used. To use your own project's output, edit TARGET_JSON_PATH in rlm_qa_agent.py.

🏗️ Project Structure

codetwine/
├── README.md
├── CHANGELOG.md                # Changelog
├── LICENSE                     # License (MIT)
├── pyproject.toml              # Package configuration and dependencies
├── main.py                     # CLI entry point
├── doc_template.json           # Design document section template definition
├── .env.example                # Environment variable template
├── codetwine/
│   ├── pipeline.py             # Main pipeline (dependency graph building → document generation → output)
│   ├── file_analyzer.py        # Per-file dependency analysis
│   ├── doc_creator.py          # Design document generation via LLM
│   ├── import_to_path.py       # Import statement to file path resolution
│   ├── output.py               # JSON and Mermaid output processing
│   ├── config/
│   │   ├── settings.py         # Environment variables and per-language settings management
│   │   └── logger.py           # Logging configuration
│   ├── extractors/
│   │   ├── definitions.py      # Definition extraction (functions, classes, etc.)
│   │   ├── imports.py          # Import statement extraction
│   │   ├── usages.py           # Symbol usage location extraction
│   │   ├── usage_analysis.py   # Usage location analysis
│   │   └── dependency_graph.py # Project-wide dependency graph construction
│   ├── parsers/
│   │   └── ts_parser.py        # Source code parser using tree-sitter
│   ├── llm/
│   │   └── client.py           # LLM API client via litellm
│   └── utils/
│       └── file_utils.py       # File operation utilities
└── examples/
    ├── doc_template_python.json  # Python-optimized design document template
    ├── rlm_qa/                   # RLM QA agent sample
    │   ├── rlm_qa_agent.py       # Interactive Q&A agent
    │   └── qa_tools.py           # Tool definitions for the agent
    └── sample_output/            # Sample output (codetwine analyzed against itself)

🙏 Acknowledgments

This project uses the following libraries:

tree-sitter - Source code syntax analysis
litellm - Unified interface for multiple LLM providers

📄 License

MIT License. See LICENSE for details.

Name		Name	Last commit message	Last commit date
Latest commit History 15 Commits
codetwine		codetwine
examples		examples
.env.example		.env.example
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
doc_template.json		doc_template.json
main.py		main.py
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Folders and files

Latest commit

History

Repository files navigation