Vibe AutoResearch

Vibe AutoResearch is a local, agent-driven research environment for improving a small two-layer nanoGPT. A Codex agent studies the results of previous runs, forms a falsifiable hypothesis, edits train.py, and launches one controlled experiment at a time. A deterministic Python orchestrator—not an LLM—runs and records each experiment.

The objective is to minimize validation bits per byte (val_bpb). Lower is better. Training uses shards 1..10; the protected validation set is shard 6542.

How it works

The project follows the OPHIS loop:

Run → Evaluate → Observe → Hypothesize → Intervene → Run

Codex may change only train.py during research. The fixed scripts prepare the data, evaluate candidates, capture observables, preserve the exact code and logs for each trial, and promote verified improvements. The first run always establishes the unmodified baseline.

Project structure

.
├── agentic/
│   ├── program.md                 # Complete Codex research protocol
│   ├── instructions/              # Git and reproducibility contracts
│   └── observables/               # Probe catalog and brainstorming workflow
├── Definition/
│   ├── AGENT.html                 # Illustrated architecture reference
│   └── assets/agent-html/         # Architecture and branch-flow images
├── frontend/                      # Live experiment dashboard
├── memory/
│   ├── read/                      # Trusted, orchestrator-written results
│   └── read_write/                # Codex hypotheses, notes, and research memory
├── scripts/
│   ├── setup_research_branch.py   # Create/check out and publish the work branch
│   ├── prepare.py                 # Fixed dataset preparation
│   ├── validation.py              # Fixed ground-truth evaluation
│   ├── observable.py              # Fixed observability helpers
│   └── update_best_branch.py      # Trusted promotion logic
├── tests/test_contract.py         # Repository contract tests
├── autoresearch_config.json       # User, compute, limits, and server settings
├── run_one_experiment.py          # Deterministic one-experiment orchestrator
├── train.py                       # The only file Codex changes during research
├── pyproject.toml                 # Python project and dependencies
└── uv.lock                        # Reproducible dependency lockfile

Each completed trial is stored below memory/read/runs/<experiment_id>/trials/t<trial_index>/, including metrics, captured output, observable data and plots, and a snapshot of the exact runnable code. memory/read/result_all.csv records every trial, while memory/read/result.csv records the baseline and accepted improvements.

Plot validation BPB over time

After experiments have produced memory/read/static/result_all.csv, generate a time-versus-validation plot from the repository root:

uv run python plot_time_vs_best_val.py

The command writes PNG and SVG plots, plus the plotted data as CSV, to artifacts/. Use --source or --output-dir to override either location.

Quick start with Codex

1. Prerequisites

• Git
• Codex CLI
• uv
• Python 3.10 or newer

A CUDA-capable machine is optional; CPU execution is supported.

2. Clone and enter the repository

git clone <repository-url>
cd vibeautoresearch

3. Configure the run

Open autoresearch_config.json and set user.name to your own short namespace. Codex will work on <name>/current; the trusted promotion script owns <name>/best. New research branches and promoted best branches are automatically pushed to origin with upstream tracking.

Choose the compute and stopping policy in the same file:

{
  "user": { "name": "your-name" },
  "run": {
    "use_cuda": false,
    "limit_mode": "runtime",
    "runtime_seconds": 300,
    "iterations": 200,
    "seed": 42
  },
  "best_verification": {
    "enabled": true,
    "trials": 10,
    "seed_end": 51
  }
}

• Set use_cuda to false for a local CPU run.
• Set it to true for local or remote CUDA; Codex will ask which target to use.
• Set limit_mode to runtime for a seconds-based limit or iterations for an optimizer-step limit.
• run.seed is the first verification seed. When verification is enabled, best_verification.seed_end is inclusive and the range size must equal best_verification.trials. The example evaluates seeds 42 through 51.
• Verification reports the mean, sample variance, and sample standard deviation of val_bpb.

Do not place remote credentials in this file. Codex stores private remote connection material only in the Git-ignored remote_device/ directory.

4. Install dependencies

uv sync

5. Start Codex

From the repository root, launch Codex:

codex

Then paste this booter prompt:

Read agentic/program.md and follow it exactly. Set up this AutoResearch session,
establish the baseline first, and then run the OPHIS experiment loop
autonomously. Continue until I interrupt you.

Codex will validate the configuration, create or check out your working branch, read the research contracts, prepare the dataset, run the baseline, analyze the trusted artifacts, and begin controlled experiments. Leave the Codex session running; the loop is designed to continue until you manually interrupt it.

Watching an experiment

While an experiment is running, open:

http://127.0.0.1:8787/frontend/

The dashboard streams training steps, validation checkpoints, observables, and the final metrics. A normal experiment is launched by Codex through:

uv run run_one_experiment.py

Do not run train.py directly: doing so bypasses the trusted orchestration and recording workflow.

Important research rules

• During experimentation, change only train.py.
• Keep the required model depth at two (DEPTH = 2).
• Never train on or otherwise leak validation shard 6542.
• Do not edit trusted files under memory/read/.
• Do not modify the fixed scripts, orchestrator, limits, or seed contract.
• Do not work directly on or force-update <name>/best.
• Commit each candidate before running it so every result maps to stable code; the orchestrator automatically pushes that commit before training starts.
• Use only dependencies already declared in pyproject.toml and uv.lock.

For the full workflow and constraints, read agentic/program.md. For the illustrated system design, open Definition/AGENT.html.