ForgePilot

A deployed AI coding agent that turns natural-language goals into validated Python projects.

ForgePilot combines LLM-based code generation, deterministic test execution, and bounded self-healing into a single workflow.

Instead of returning the first code generated by an LLM, ForgePilot generates a project, runs its tests, and—if validation fails—uses the failure output to attempt one targeted repair.

LLM for generation. Tests for verification.

Live

Web Application

https://forgepilot001.vercel.app

REST API

https://forgepilot-my6c.onrender.com

Generation Endpoint

POST /api/generate

Why ForgePilot?

LLMs are capable of producing surprisingly complete software from natural-language descriptions, but generated code is not guaranteed to work.

ForgePilot treats code generation as a:

Generate → Verify → Repair

pipeline.

A generated project must contain at least one test_*.py file. The project is written into an isolated temporary workspace and validated using pytest.

If the tests fail, the validation output is sent to a self-healing LLM pass. The repaired project is then validated again.

This creates a clear separation of responsibilities:

Probabilistic
     LLM
     │
     │ generates / repairs
     ▼
Deterministic
   pytest
     │
     │ verifies
     ▼
   Result

How It Works

flowchart TD
    A[User Goal] --> B[FastAPI /api/generate]
    B --> C[LLM Budget: 2 Calls Max]
    C --> D[Project Generation]
    D --> E[Temporary Isolated Workspace]
    E --> F[pytest Validation]

    F -->|Pass| G[Return Generated Files]
    F -->|Fail| H[Self-Healing LLM]
    H --> I[Rewrite Workspace]
    I --> J[pytest Validation Again]

    J -->|Pass| K[Return Healed Files]
    J -->|Fail| L[Return Validation Error]

Request lifecycle

Receive the goal through the FastAPI endpoint.
Create a two-call LLM budget for the request.
Generate the project as a structured files mapping.
Create an isolated temporary workspace for that request.
Write the generated files into the workspace.
Run pytest against the generated project.
If validation succeeds, return the project immediately.
If validation fails, pass the original goal and validation error to the self-healing agent.
Replace the failed project with the repaired files.
Run validation again.
Return either the validated project or a 422 error containing the validation failure.

Key Engineering Decisions

Bounded LLM Usage

Every generation request receives a hard budget of two LLM calls:

Call 1 → Initial generation
Call 2 → Self-healing, only if validation fails

There is no unbounded retry loop.

This keeps inference behavior predictable and prevents repeated repair attempts from silently increasing cost.

Deterministic Verification

ForgePilot does not ask the model whether its own code works.

Generated projects are tested using:

python -m pytest

The validation result is determined by the test process rather than another LLM judgment.

Traceback-Driven Self-Healing

When the first validation fails, ForgePilot passes the actual validation output to the repair model:

Original goal
     +
pytest failure
     ↓
Self-healing LLM
     ↓
Repaired project

The repair step therefore has concrete failure information instead of simply regenerating the project without context.

Request-Level Workspace Isolation

Each API request receives its own temporary directory.

Generated files are materialized there, validated, and discarded when the request finishes.

This prevents one request's generated workspace from being reused by another request.

Conditional Repair

The second LLM call is only made when the first validation fails.

A project that passes on the first attempt therefore requires only one generation call.

API

`POST /api/generate`

Generate and validate a project from a natural-language goal.

Request

{
  "goal": "Build a Python calculator with tests"
}

First-Pass Success

{
  "status": "success",
  "healed": false,
  "files": {
    "calculator.py": "...",
    "test_calculator.py": "..."
  }
}

Self-Healed Success

{
  "status": "success",
  "healed": true,
  "files": {
    "calculator.py": "...",
    "test_calculator.py": "..."
  }
}

The healed field indicates whether the generated project passed validation immediately or required a repair attempt.

Error Responses

400 — Invalid Goal

Returned when the supplied goal is empty.

422 — Validation Failure

Returned when the generated project fails validation and the repair attempt also fails.

500 — Generation Failure

Returned when the initial generation process itself fails.

Generation Contract

ForgePilot asks the LLM to return a structured project rather than prose or Markdown.

The core representation is:

{
  "files": {
    "app.py": "...",
    "utils.py": "...",
    "test_app.py": "..."
  }
}

Every generated project must include at least one:

test_*.py

For web and UI-oriented goals, the generation pipeline additionally instructs the model to include the required HTML/CSS/JavaScript assets and tests capable of validating the generated interface.

Project Structure

ForgePilot/
│
├── app.py                    # FastAPI application and API endpoint
├── cli.py                    # Original CLI interface
├── config.py                 # Configuration and environment loading
├── requirements.txt
│
├── agent/
│   ├── budget.py             # Per-request LLM call budget
│   ├── cache.py              # CLI caching / last-known-good support
│   ├── generator.py          # Initial project generation
│   ├── json_utils.py         # Structured LLM output handling
│   ├── llm.py                # LLM integration
│   ├── llm_json.py           # Structured file generation
│   ├── normalize.py          # Goal normalization
│   ├── prompts.py            # Generation and self-healing prompts
│   ├── self_heal.py          # Validation-driven repair
│   └── validator.py          # pytest execution
│
├── frontend/
│   └── index.html            # Web frontend
│
└── workspace/
    └── ...                   # Example/generated project files

Architecture

The deployed version separates the user-facing frontend from the generation API:

┌─────────────────────────┐
│     Vercel Frontend     │
│      Web Interface      │
└────────────┬────────────┘
             │ HTTP
             ▼
┌─────────────────────────┐
│      Render Backend     │
│        FastAPI          │
└────────────┬────────────┘
             │
             ▼
┌─────────────────────────┐
│      ForgePilot Agent   │
├─────────────────────────┤
│ Generation              │
│ Validation              │
│ Self-Healing            │
│ LLM Budget              │
└─────────────────────────┘

The original CLI interface remains available through cli.py for local experimentation.

Local Development

Prerequisites

Python 3.10+
Google AI API key with access to the configured Gemini model
Network access for LLM calls

1. Clone

git clone https://github.com/iamzimozic/ForgePilot.git
cd ForgePilot

2. Create a virtual environment

python3 -m venv .venv
source .venv/bin/activate

On Windows:

.venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

4. Configure the API key

Create a .env file:

GOOGLE_API_KEY=your_google_api_key_here

Do not commit .env or real API keys.

5. Start the API

uvicorn app:app --reload

The API will be available at:

http://localhost:8000

6. Generate a project

curl -X POST http://localhost:8000/api/generate \
  -H "Content-Type: application/json" \
  -d '{"goal":"Build a Python calculator with tests"}'

CLI

ForgePilot also retains its original CLI interface.

Example:

python cli.py "Build a CLI that converts Celsius and Fahrenheit"

Other examples:

python cli.py "Create a minimal FastAPI app with a /health endpoint and tests"

python cli.py "Write a tiny library for slugifying strings with pytest"

The CLI additionally contains the project's local caching and last-known-good workflow.

Caching

The CLI supports deterministic caching based on a normalized goal.

Normalization:

converts text to lowercase
strips leading/trailing whitespace
collapses repeated internal whitespace

The normalized goal is hashed using SHA-256 to produce the cache key.

For example:

Create a FastAPI App

and:

create   a   fastapi app

map to the same cache key.

A successful cached result can be reused without another LLM call.

The cache also maintains a last-known-good result for the CLI fallback path.

The deployed FastAPI request path uses the bounded generation/self-healing pipeline described above. The CLI cache is a separate local optimization.

LLM Budget

ForgePilot deliberately limits inference per generation request.

The deployed API creates:

LLMBudget(limit=2)

The two possible calls are:

1. generate_project()
2. self_heal()       # only after validation failure

If the first generated project passes validation, only one LLM call is required.

Validation

Generated projects are required to contain at least one test_*.py file.

Validation:

Finds the generated test files.
Executes pytest inside the generated workspace.
Captures stdout and stderr.
Returns the validation output to the self-healing stage when validation fails.

The validation command is:

python -m pytest

The test process itself acts as the deterministic gate for generated output.

Security Notes

ForgePilot is a development-oriented AI coding system and should not be treated as a hardened sandbox for arbitrary untrusted code.

Important considerations:

Generated code is executed during validation.
Temporary directories provide request-level filesystem isolation, but are not a complete security boundary.
Production deployments should use stronger execution isolation, resource limits, and process/container restrictions before accepting untrusted workloads.
Keep GOOGLE_API_KEY in environment variables or a secret manager.
Never commit real credentials.
Review generated code before deploying generated applications.

Current Limitations

ForgePilot is intentionally constrained.

Python-focused project generation
Generated projects must include pytest tests
One self-healing attempt per request
No unrestricted multi-step repair loop
No incremental editing of an existing repository
No persistent project workspace in the current API flow
No authentication or user-specific project history in the current API
Temporary workspaces are not equivalent to hardened code-execution sandboxes
Generated project structure is constrained by the generation prompts
Model availability depends on the configured Gemini API/model

Future Directions

Potential improvements include:

Stronger code-execution sandboxing
Containerized or isolated execution workers
Resource and time limits for generated programs
Incremental repository editing
Multi-step bounded repair strategies
Persistent project workspaces
Authentication and project history
Larger automated evaluation suites
Support for additional languages and toolchains
More granular observability and cost tracking

Tech Stack

Backend

Python
FastAPI
Pydantic

AI

Gemini
LangChain
Structured LLM output
Traceback-driven self-healing

Validation

pytest

Frontend

HTML
CSS
JavaScript

Deployment

Vercel
Render

Project Philosophy

ForgePilot is built around a simple idea:

Code generation is not the same thing as working software.

An LLM can propose an implementation.

A test suite can tell you whether that implementation actually works.

ForgePilot puts those two pieces into a bounded feedback loop:

Generate
   ↓
Execute
   ↓
Validate
   ↓
Repair if necessary
   ↓
Validate again
   ↓
Return

The goal is not to make the LLM blindly retry until something works.

The goal is to make the LLM operate inside an explicit engineering feedback loop with deterministic verification and bounded inference.

License

No license file is currently included in this repository.

Name		Name	Last commit message	Last commit date
agent		agent
frontend		frontend
workspace		workspace
.gitignore		.gitignore
README.md		README.md
app.py		app.py
cli.py		cli.py
config.py		config.py
repomix-output.xml		repomix-output.xml
requirements.txt		requirements.txt

Latest commit

History

Folders and files

Repository files navigation