AI Research Agent Benchmarking Workspace

This repository is a research and benchmarking project for evaluating how well language models generate correct software from task specifications. It is not a single end-user app; it is a harness for generating code, writing benchmark runs to timestamped directories, and comparing model quality across task difficulty, prompting styles, and evaluation conditions.

The project supports:

• zero-shot generation
• RAG-assisted generation using a local retrieval layer
• human baseline references
• deterministic validation and report generation
• comparisons across Gemini, Llama, Qwen, and DeepSeek-backed runs

---

Overview

The repo generates projects from natural-language tasks, writes them into run-specific workspace folders under directories such as runs/, runs_new_01/, runs_new_02/, runs_rag/, and runs_zero_01/, then evaluates them with pytest and coverage-related metrics.

Typical workflow:

1. A task definition is created in a generator script.
2. A model backend is selected from the configured list.
3. The model generates a complete flat project as JSON ({"files": {...}}).
4. Files are written into a timestamped workspace/ under a run directory.
5. The generated project is evaluated with deterministic tests and/or benchmark checks.
6. Result rows are collected into CSV outputs for comparison and plotting.

This makes the repo useful for code-generation benchmarking, not for shipping one fixed app.

---

What the project does now

The current codebase is centered around research experiments comparing model behavior across a set of benchmark tasks.

Challenge levels

The benchmark tasks are grouped into:

• simple
• intermediate
• complex

Examples include:

• Fibonacci implementation and parsing utilities
• CSV summarization and data-processing functions
• FastAPI CRUD apps
• async RAG pipelines
• web scraping wrappers
• database-backed inventory logic
• metadata alignment pipelines
• Django-style app scaffolds

Baselines and variants

The repo includes several benchmark variants:

• generate.py — standard model generation benchmark
• generate_with_rag.py — generation with a local RAG retrieval engine
• coderag-generate.py and coderag-generate_rag.py — alternate benchmark entry points
• evaluate_tests.py — evaluates generated workspaces across runs
• evaluate_rag.py — evaluates RAG benchmark outputs
• evaluate_zero.py — evaluates zero-shot benchmark outputs
• human_baseline/ — reference implementations and task-specific tests

---

Repository structure

ai_research_agent/
├── agent/
│   ├── budget.py
│   ├── cache.py
│   ├── generator.py
│   ├── json_utils.py
│   ├── llm.py
│   ├── normalize.py
│   ├── prompts.py
│   ├── prompts_new.py
│   ├── self_heal.py
│   ├── structured_output.py
│   ├── validator.py
│   └── ...
├── deterministic_tests/
│   ├── test_csv_json.py
│   ├── test_fast.py
│   ├── test_fibonacci.py
│   ├── test_json_utils.py
│   ├── test_sensor_utils.py
│   ├── test_sequence.py
│   └── ...
├── human_baseline/
│   ├── simple/
│   ├── intermediate/
│   └── complex/
├── runs/
├── runs_after/
├── runs_new_01/
├── runs_new_02/
├── runs_rag/
├── runs_rag_01/
├── runs_zero_01/
├── save_files/
│   ├── benchmark.py
│   ├── deep_eval.py
│   ├── evaluate_saved_tasks.py
│   ├── generate_01.py
│   └── ...
├── visualizations/
├── workspace/
├── cli.py
├── config.py
├── generate.py
├── generate_with_rag.py
├── coderag-generate.py
├── coderag-generate_rag.py
├── evaluate_tests.py
├── evaluate_rag.py
├── evaluate_zero.py
├── visualize_results.py
├── visualize_results_01.py
├── visualize_results_W.py
├── requirements.txt
├── README.md
└── ...

Key points:

• config.py sets model names and runtime constants.
• agent/llm.py swaps between Gemini and local Ollama-backed model families.
• agent/generator.py handles generation requests.
• agent/self_heal.py attempts repair when the generated code fails validation.
• runs_* directories store output workspaces for each model and task.
• results_*.csv files are benchmark outputs used for analysis.

---

Model and backend support

The current platform supports multiple backends through the config and LLM wrapper layer.

Configured defaults in config.py include:

• GEMINI_MODEL = "gemini-2.5-flash"
• LLAMA_MODEL = "llama3.1:8b"
• QWEN_MODEL = "qwen2.5-coder:7b"
• DEEPSEEK_MODEL = "deepseek-coder:6.7b"

The LLM factory in agent/llm.py routes model names to either:

• ChatGoogleGenerativeAI for Gemini
• ChatOllama for local Llama/Qwen/DeepSeek-style models

This is used by scripts such as generate.py and the RAG generation variants.

---

Core benchmark flow

1. Task generation

Scripts such as generate.py define a set of benchmark tasks, each with a task level and goal description. Each run creates a fresh timestamped workspace directory.

Example task types:

• simple_fibonacci
• simple_log_parser
• simple_matrix_math
• intermediate_financial_csv
• intermediate_web_scraper
• complex_fastapi_crud
• complex_async_rag

2. Generation and validation

The generation pipeline writes all files for a project into a single directory with a flat layout. The agent expects each project to include at least one test_*.py file and validates with pytest.

This behavior is enforced by prompts in:

• agent/prompts.py
• agent/prompts_new.py

and by validation code in:

• agent/validator.py

3. Benchmark scoring

Evaluation scripts run tests, collect coverage, and save structured summaries to CSV. For example:

• evaluate_tests.py scans workspace directories and records pass/fail states
• evaluate_rag.py measures RAG-benchmark performance
• evaluate_zero.py measures zero-shot benchmark performance

The evaluation scripts also capture coverage and basic reliability metrics similar to pass@1, test counts, and statement coverage.

---

Quick start

Install dependencies

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

For evaluations and tests, ensure the required Python tooling is present.

Configure the API key

If using Gemini, set the environment variable:

export GOOGLE_API_KEY="your_api_key_here"

The project also supports local model runs if Ollama and compatible model names are available.

Run a standard generation sweep

python generate.py --tasks all --models all

This will create run outputs under runs/ for the configured task dimensions and model list.

Run a RAG-enabled comparison

python generate_with_rag.py

This uses a local vector store and RAG retrieval layer before generation.

Evaluate generated workspaces

python evaluate_tests.py --runs-dir runs_new_01 --output results_main_01.csv

or:

python evaluate_rag.py
python evaluate_zero.py

---

Output directories and artifacts

The project produces several types of outputs:

• runs/ — baseline model runs
• runs_new_01/, runs_new_02/ — later benchmark sweeps
• runs_rag/, runs_rag_01/ — RAG run outputs
• runs_zero_01/ — zero-shot run outputs
• human_baseline/ — human reference solutions
• results_*.csv — aggregate benchmark outputs
• visualizations/ and plotting scripts for comparing metrics

Each run usually follows a structure like:

runs_new_01/
└── complex/
    └── gemini-2.5-flash/
        └── 20260605T123456Z/
            └── workspace/
                ├── main.py
                ├── test_main.py
                └── requirements.txt

---

Benchmark tasks and evaluation philosophy

The repo is designed to answer research questions like:

• Which model is most reliable on structured programming tasks?
• How does RAG change correctness and coverage?
• How do different task complexities affect model performance?
• Does generated code fail due to logic bugs, missing tests, or execution issues?

The benchmark therefore emphasizes:

• correctness under pytest
• task difficulty tiers
• model-to-model comparison
• coverage and pass/fail summaries
• repeatable run directories for post-hoc analysis

---

Important notes

• This repo is a benchmarking project, not a production application generator for one fixed product.
• Generated workspaces are intentionally stored in timestamped, task-specific directories.
• The code expects flat directory layouts and Python test files named like test_*.py.
• The project is designed to compare multiple model families under controlled conditions.
• workspace/ at the repo root is a mutable scratch area and not the canonical benchmark output location.

---

Common commands

python generate.py --tasks simple --models gemini
python generate.py --tasks all --models all
python generate_with_rag.py
python evaluate_tests.py --runs-dir runs_new_01 --output results_main_01.csv
python visualize_results.py

---

Development notes

The current repo contains both the benchmarking harness and the model-generation infrastructure that feeds it. The main extension points are:

• config.py — model selection and runtime constants
• agent/prompts.py and agent/prompts_new.py — generation and repair instruction templates
• agent/llm.py — backend routing
• evaluate_*.py — result aggregation and scoring
• visualize_results*.py — plotting and comparison

---

License

No explicit license file is included in this repository.