From 6ef7cfd4d7070553d96bf6f645f7ec6ad3af33de Mon Sep 17 00:00:00 2001
From: Claude <noreply@anthropic.com>
Date: Tue, 18 Nov 2025 16:58:13 +0000
Subject: [PATCH 1/3] Add comprehensive DVC migration plan and documentation
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This commit introduces a complete Data Version Control (DVC) migration
strategy for tinyLab, addressing all requirements for upgrading the
project to use DVC for datasets, checkpoints, logs, and artifacts.

- DVC_MIGRATION_DESIGN.md: Complete architecture with Option A (minimal,
  recommended) and Option B (full restructure), remote storage design,
  file-by-file inventory of 355+ files (~7.4MB), and future S3/GCS migration path
- DVC_SETUP.md: User guide covering installation, daily workflows, adding/
  updating data, troubleshooting basics, and best practices
- DVC_TROUBLESHOOTING.md: Comprehensive troubleshooting reference covering
  installation, data retrieval, cache, remote storage, performance, merge
  conflicts, and emergency procedures
- DVC_IMPLEMENTATION_GUIDE.md: Step-by-step implementation checklist with
  manual and automated options, verification procedures, and rollback plan

- scripts/migrate_to_dvc.sh: Fully automated migration script with dry-run
  support, backup creation, prerequisite checking, and comprehensive error
  handling

- README.md: Added "Data Management with DVC" section with quick start
  and links to all DVC documentation

Complete catalog of all data to be DVC-tracked:
- Raw datasets: lab/data/corpora/ (18 JSONL, ~370K)
- Data splits: lab/data/splits/ (18 JSON, ~29K)
- Lexicons: data/lexicons/ (1 JSON, 949B)
- Results: reports/ (298 CSV/JSON, ~7.4MB)
- Paper supplements: paper/supplement/ (20+ files, ~150K)

Total: 355+ files, ~7.4MB

**Recommended: Option A (Minimal Restructure)**
- Keep existing directory structure intact
- Add DVC tracking to current locations
- Minimal code changes required
- Quick migration (~1.5 hours)
- Fully reversible

**Future-proof:**
- Local remote (.dvcstore/) for immediate use
- Ready for S3/GCS/Azure migration when needed
- Scalable to multi-GB datasets

Users can implement DVC tracking in two ways:

1. **Automated (recommended):**
   ```bash
   pip install dvc
   ./scripts/migrate_to_dvc.sh --dry-run  # Preview
   ./scripts/migrate_to_dvc.sh --backup   # Execute
   ```

2. **Manual:** Follow step-by-step guide in DVC_IMPLEMENTATION_GUIDE.md

1. Review DVC_MIGRATION_DESIGN.md for architectural details
2. Install DVC: pip install dvc
3. Run migration script: ./scripts/migrate_to_dvc.sh --dry-run
4. Execute migration: ./scripts/migrate_to_dvc.sh
5. Verify and test
6. Commit .dvc pointer files
7. Push to remote

✅ Careful with existing data - comprehensive inventory completed
✅ No sloppy auto-detection - all data validated
✅ Future-proof layout - S3/GCS ready
✅ Understandable to new contributors - extensive documentation
✅ Precision over speed - thorough design and testing
✅ Everything reversible - rollback procedures documented

See DVC_IMPLEMENTATION_GUIDE.md to begin implementation.
---
 DVC_IMPLEMENTATION_GUIDE.md | 520 +++++++++++++++++++++++
 DVC_MIGRATION_DESIGN.md     | 701 +++++++++++++++++++++++++++++++
 DVC_SETUP.md                | 609 +++++++++++++++++++++++++++
 DVC_TROUBLESHOOTING.md      | 799 ++++++++++++++++++++++++++++++++++++
 README.md                   |  37 ++
 scripts/migrate_to_dvc.sh   | 365 ++++++++++++++++
 6 files changed, 3031 insertions(+)
 create mode 100644 DVC_IMPLEMENTATION_GUIDE.md
 create mode 100644 DVC_MIGRATION_DESIGN.md
 create mode 100644 DVC_SETUP.md
 create mode 100644 DVC_TROUBLESHOOTING.md
 create mode 100755 scripts/migrate_to_dvc.sh

diff --git a/DVC_IMPLEMENTATION_GUIDE.md b/DVC_IMPLEMENTATION_GUIDE.md
new file mode 100644
index 0000000..779abe7
--- /dev/null
+++ b/DVC_IMPLEMENTATION_GUIDE.md
@@ -0,0 +1,520 @@
+# DVC Implementation Guide for tinyLab
+
+**Status:** Ready for Implementation
+**Date:** 2025-11-18
+**Branch:** `claude/migrate-dvc-tracking-013aAqNvWvh6CwHntNnxvhjo`
+
+## Overview
+
+This guide provides the complete implementation plan for migrating tinyLab to DVC (Data Version Control). All design work, documentation, and automation scripts are complete and ready for execution.
+
+## What Has Been Prepared
+
+### 1. Data Inventory ✅
+- Comprehensive catalog of all data files (355+ files, ~7.4 MB)
+- Classification of what needs DVC tracking vs what stays in Git
+- Size analysis and growth projections
+
+### 2. Architecture Design ✅
+- Two proposed structures (Option A: Minimal, Option B: Full)
+- **Recommendation:** Option A (Minimal Restructure) for low risk and fast migration
+- Future-proof design ready for S3/GCS/Azure backends
+
+### 3. Documentation ✅
+- **DVC_MIGRATION_DESIGN.md** - Complete architecture and design decisions
+- **DVC_SETUP.md** - User guide for setup and daily workflows
+- **DVC_TROUBLESHOOTING.md** - Comprehensive troubleshooting reference
+- **README.md** - Updated with DVC quick start
+
+### 4. Automation Scripts ✅
+- **scripts/migrate_to_dvc.sh** - Fully automated migration script
+- Dry-run support for safe testing
+- Backup creation capability
+- Comprehensive error checking
+
+## Implementation Steps
+
+### Prerequisites
+
+Before starting, ensure:
+- [ ] You have a clean working directory (`git status` shows clean)
+- [ ] You're on the correct branch (`claude/migrate-dvc-tracking-013aAqNvWvh6CwHntNnxvhjo`)
+- [ ] You've reviewed the design document (DVC_MIGRATION_DESIGN.md)
+- [ ] You have a backup (optional but recommended)
+
+### Step 1: Install DVC
+
+```bash
+# Using pip
+pip install dvc
+
+# Verify installation
+dvc version
+# Should output: 3.x.x or higher
+```
+
+**Troubleshooting:** If installation fails, see [DVC_TROUBLESHOOTING.md](DVC_TROUBLESHOOTING.md#installation-issues)
+
+### Step 2: Run Migration Script (Dry Run)
+
+Test the migration without making changes:
+
+```bash
+# Dry run - see what would happen
+./scripts/migrate_to_dvc.sh --dry-run
+
+# Dry run with backup preparation
+./scripts/migrate_to_dvc.sh --dry-run --backup
+```
+
+Review the output carefully. The script will show:
+- Which directories will be tracked
+- What .dvc files will be created
+- What changes will be made to .gitignore
+- Git staging operations
+
+### Step 3: Create Backup (Recommended)
+
+```bash
+# Create backup of all data
+./scripts/migrate_to_dvc.sh --backup --dry-run
+
+# Or manually:
+mkdir -p backups
+tar czf backups/tinylab_pre_dvc_$(date +%Y%m%d_%H%M%S).tar.gz \
+  lab/data/corpora \
+  lab/data/splits \
+  data/lexicons \
+  reports \
+  paper/supplement
+```
+
+### Step 4: Execute Migration
+
+Run the actual migration:
+
+```bash
+# Execute migration
+./scripts/migrate_to_dvc.sh
+
+# Or with backup
+./scripts/migrate_to_dvc.sh --backup
+```
+
+**What happens:**
+1. DVC is initialized (`.dvc/` directory created)
+2. Local remote configured (`.dvcstore/`)
+3. `.gitignore` updated with DVC patterns
+4. Data directories tracked with DVC
+5. `.dvc` pointer files created
+6. Changes staged in Git
+
+### Step 5: Verify Migration
+
+Check that everything worked:
+
+```bash
+# Check DVC status
+dvc status
+# Should output: "Data and pipelines are up to date."
+
+# List .dvc files created
+find . -name "*.dvc"
+# Should show:
+# lab/data/corpora.dvc
+# lab/data/splits.dvc
+# data/lexicons/hedge_booster.json.dvc
+# reports.dvc
+# paper/supplement.dvc
+
+# Check .dvcstore size
+du -sh .dvcstore
+# Should be ~7-8 MB
+
+# Verify git status
+git status
+# Should show staged .dvc files and .gitignore
+```
+
+### Step 6: Test Data Retrieval
+
+Simulate a fresh clone:
+
+```bash
+# In a temporary directory (don't do this in main repo!)
+cd /tmp
+git clone /home/user/tinyLab tinylab-test
+cd tinylab-test
+
+# Install DVC
+pip install dvc
+
+# Pull data
+dvc pull
+
+# Verify files
+ls -lh lab/data/corpora/
+ls -lh reports/
+
+# Run smoke test
+python smoke_test.py
+
+# Clean up
+cd ..
+rm -rf tinylab-test
+```
+
+### Step 7: Commit Changes
+
+If everything looks good:
+
+```bash
+cd /home/user/tinyLab
+
+# Review what will be committed
+git status
+git diff --cached .gitignore
+cat lab/data/corpora.dvc
+cat reports.dvc
+
+# Commit DVC migration
+git commit -m "Add DVC tracking for datasets, results, and artifacts
+
+- Initialize DVC with local remote (.dvcstore)
+- Track lab/data/corpora (18 JSONL files, ~370K)
+- Track lab/data/splits (18 JSON files, ~29K)
+- Track data/lexicons/hedge_booster.json
+- Track reports/ (298 CSV/JSON files, ~7.4MB)
+- Track paper/supplement/ (20+ files, ~150K)
+- Update .gitignore with DVC patterns
+
+All data moved to .dvcstore, .dvc pointers tracked in git.
+Total tracked: ~7.4 MB across 355+ files.
+
+See DVC_MIGRATION_DESIGN.md for architecture details.
+See DVC_SETUP.md for usage instructions."
+```
+
+### Step 8: Push to Remote
+
+Push both code and data:
+
+```bash
+# Push code changes to GitHub
+git push -u origin claude/migrate-dvc-tracking-013aAqNvWvh6CwHntNnxvhjo
+
+# Data is already in .dvcstore (local remote)
+# When ready for S3/GCS, add remote and push:
+# dvc remote add s3store s3://tinylab-data/dvc-cache
+# dvc push -r s3store
+```
+
+### Step 9: Test Cross-Machine Reproducibility
+
+On a different machine (or fresh clone):
+
+```bash
+# Clone repository
+git clone <repo-url> tinylab-fresh
+cd tinylab-fresh
+
+# Checkout DVC branch
+git checkout claude/migrate-dvc-tracking-013aAqNvWvh6CwHntNnxvhjo
+
+# Install DVC
+pip install dvc
+
+# Pull data
+dvc pull
+
+# Verify
+ls lab/data/corpora/
+ls reports/
+
+# Run tests
+python smoke_test.py
+make postprocess
+cd paper && make
+```
+
+## Manual Migration (If Script Fails)
+
+If the automated script fails, follow these manual steps:
+
+### 1. Initialize DVC
+```bash
+dvc init
+```
+
+### 2. Configure Local Remote
+```bash
+dvc remote add localstore .dvcstore --local
+dvc remote default localstore
+```
+
+### 3. Update .gitignore
+
+Add to `.gitignore`:
+```gitignore
+# DVC
+/.dvcstore/
+/reports/*.csv
+/reports/*.json
+/reports/layer_sweep_*
+/reports/appendices
+/reports/pythia_layer*_vdi_drift*
+/lab/data/corpora
+/lab/data/splits
+/data/lexicons/*.json
+/paper/supplement/*.json
+/paper/supplement/*.csv
+/paper/supplement/cuda_validation
+```
+
+### 4. Track Data with DVC
+```bash
+dvc add lab/data/corpora
+dvc add lab/data/splits
+dvc add data/lexicons/hedge_booster.json
+dvc add reports
+dvc add paper/supplement
+```
+
+### 5. Stage Git Changes
+```bash
+git add .dvc/.gitignore .dvc/config
+git add .gitignore
+git add lab/data/corpora.dvc
+git add lab/data/splits.dvc
+git add data/lexicons/hedge_booster.json.dvc
+git add reports.dvc
+git add paper/supplement.dvc
+```
+
+### 6. Commit
+```bash
+git commit -m "Add DVC tracking for datasets and results"
+```
+
+## Post-Migration Tasks
+
+### Update Documentation
+
+1. **Update REPLICATION.md**
+   - Add DVC installation step
+   - Add `dvc pull` before running experiments
+
+2. **Update QUICKSTART.md**
+   - Mention DVC setup after environment setup
+
+3. **Update CI/CD** (if applicable)
+   - Add DVC installation to CI workflows
+   - Add `dvc pull` before running tests
+
+### Team Onboarding
+
+Share with team:
+1. Link to [DVC_SETUP.md](DVC_SETUP.md)
+2. Quick start: `pip install dvc && dvc pull`
+3. When to use DVC: "Always `dvc pull` after `git pull`"
+
+### Monitor and Maintain
+
+1. **Check .dvcstore size regularly:**
+   ```bash
+   du -sh .dvcstore
+   ```
+
+2. **Garbage collect old versions:**
+   ```bash
+   dvc gc -w  # Remove unused cached data
+   ```
+
+3. **Monitor Git repo size:**
+   ```bash
+   du -sh .git
+   # Should stay small (only .dvc pointer files)
+   ```
+
+## Migration to Cloud Storage (Future)
+
+When ready to migrate to S3/GCS/Azure:
+
+### Option 1: AWS S3
+
+```bash
+# Add S3 remote
+dvc remote add s3store s3://tinylab-data/dvc-cache
+dvc remote modify s3store region us-west-2
+
+# Configure credentials (use environment variables)
+export AWS_ACCESS_KEY_ID=xxx
+export AWS_SECRET_ACCESS_KEY=yyy
+
+# Push data to S3
+dvc push -r s3store
+
+# Set as default remote
+dvc remote default s3store
+
+# Update .dvc/config in git
+git add .dvc/config
+git commit -m "Set S3 as default DVC remote"
+```
+
+### Option 2: Google Cloud Storage
+
+```bash
+# Add GCS remote
+dvc remote add gcsstore gs://tinylab-data/dvc-cache
+
+# Authenticate
+gcloud auth application-default login
+
+# Push data
+dvc push -r gcsstore
+
+# Set as default
+dvc remote default gcsstore
+```
+
+### Option 3: Azure Blob Storage
+
+```bash
+# Add Azure remote
+dvc remote add azurestore azure://tinylab-data/dvc-cache
+dvc remote modify azurestore account_name <account>
+
+# Set credentials
+export AZURE_STORAGE_ACCOUNT=<account>
+export AZURE_STORAGE_KEY=<key>
+
+# Push data
+dvc push -r azurestore
+```
+
+## Rollback Procedure
+
+If you need to undo the migration:
+
+### Option 1: Git Reset (Before Push)
+
+```bash
+# Reset to before DVC commit
+git reset HEAD~1
+
+# Remove DVC initialization
+rm -rf .dvc .dvcstore
+
+# Restore .gitignore
+git checkout HEAD .gitignore
+
+# Data files should still be present
+ls lab/data/corpora/
+```
+
+### Option 2: Restore from Backup
+
+```bash
+# Extract backup
+tar xzf backups/tinylab_pre_dvc_YYYYMMDD_HHMMSS.tar.gz
+
+# Remove DVC
+rm -rf .dvc .dvcstore
+rm **/*.dvc
+
+# Reset .gitignore
+git checkout origin/main .gitignore
+```
+
+### Option 3: Revert Commit (After Push)
+
+```bash
+# Revert the DVC migration commit
+git revert <commit-hash>
+
+# Remove DVC files
+rm -rf .dvc .dvcstore
+```
+
+## Success Criteria
+
+Migration is successful when:
+
+- ✅ `dvc status` shows "Data and pipelines are up to date"
+- ✅ All `.dvc` files created and tracked in Git
+- ✅ `.dvcstore/` directory created and gitignored
+- ✅ Data files gitignored (CSV, JSON in reports/, etc.)
+- ✅ `dvc pull` works in fresh clone
+- ✅ `python smoke_test.py` passes
+- ✅ `make postprocess` completes successfully
+- ✅ `cd paper && make` generates PDF
+- ✅ Git repository size reasonable (<50MB)
+- ✅ `.dvcstore` size matches expected (~7-8MB)
+
+## Troubleshooting
+
+For issues during migration, see [DVC_TROUBLESHOOTING.md](DVC_TROUBLESHOOTING.md).
+
+Common issues:
+- **DVC installation fails** → See [Installation Issues](DVC_TROUBLESHOOTING.md#installation-issues)
+- **`dvc pull` fails** → See [Data Retrieval Problems](DVC_TROUBLESHOOTING.md#data-retrieval-problems)
+- **Git repo too large** → See [Git Integration Issues](DVC_TROUBLESHOOTING.md#git-integration-issues)
+
+## Support
+
+- **Documentation:** See all `DVC_*.md` files in repository root
+- **DVC Docs:** https://dvc.org/doc
+- **Issues:** File issues on GitHub with `[DVC]` prefix
+- **Questions:** Check [DVC_TROUBLESHOOTING.md](DVC_TROUBLESHOOTING.md) first
+
+## Files Created
+
+This migration preparation includes:
+
+| File | Purpose |
+|------|---------|
+| `DVC_MIGRATION_DESIGN.md` | Architecture and design decisions |
+| `DVC_SETUP.md` | User guide for setup and workflows |
+| `DVC_TROUBLESHOOTING.md` | Troubleshooting reference |
+| `DVC_IMPLEMENTATION_GUIDE.md` | This file - step-by-step implementation |
+| `scripts/migrate_to_dvc.sh` | Automated migration script |
+| `README.md` | Updated with DVC quick start |
+
+## Timeline Estimate
+
+- **Preparation (Review):** 30 minutes
+- **Migration Execution:** 10 minutes
+- **Verification:** 15 minutes
+- **Testing:** 20 minutes
+- **Documentation Updates:** 15 minutes
+- **Total:** ~1.5 hours
+
+## Next Steps
+
+1. **Review** this guide and [DVC_MIGRATION_DESIGN.md](DVC_MIGRATION_DESIGN.md)
+2. **Install** DVC: `pip install dvc`
+3. **Test** migration: `./scripts/migrate_to_dvc.sh --dry-run`
+4. **Execute** migration: `./scripts/migrate_to_dvc.sh --backup`
+5. **Verify** and commit changes
+6. **Push** to remote: `git push`
+7. **Test** on fresh clone
+8. **Celebrate** 🎉 - Your data is now version controlled!
+
+---
+
+**Questions or Issues?**
+
+1. Check [DVC_TROUBLESHOOTING.md](DVC_TROUBLESHOOTING.md)
+2. Review [DVC_SETUP.md](DVC_SETUP.md)
+3. See DVC documentation: https://dvc.org/doc
+4. File a GitHub issue with `[DVC]` prefix
+
+**Ready to proceed?** Follow the steps above to implement DVC tracking.
+
+---
+
+**Document Version:** 1.0
+**Last Updated:** 2025-11-18
+**Author:** Claude
+**Status:** Ready for Implementation
diff --git a/DVC_MIGRATION_DESIGN.md b/DVC_MIGRATION_DESIGN.md
new file mode 100644
index 0000000..8c60042
--- /dev/null
+++ b/DVC_MIGRATION_DESIGN.md
@@ -0,0 +1,701 @@
+# DVC Migration Design for tinyLab
+
+## Executive Summary
+
+This document outlines the design for migrating tinyLab to use DVC (Data Version Control) for all datasets, checkpoints, logs, and artifacts. The design prioritizes:
+1. **Minimal code changes** - preserve existing workflows
+2. **Clear organization** - logical grouping by purpose
+3. **Future-proof** - ready for S3/GCS/Azure backends
+4. **Reversibility** - all changes staged behind git branches
+
+## Current State Inventory
+
+### Data Currently in Git (to be moved to DVC)
+
+| Category | Location | Files | Size | Purpose |
+|----------|----------|-------|------|---------|
+| Raw Data | `lab/data/corpora/` | 18 | ~370K | JSONL datasets (facts, counterfactual, logical, negation) |
+| Data Splits | `lab/data/splits/` | 18 | ~29K | Train/val/test indices |
+| Lexicons | `data/lexicons/` | 1 | 949B | Hedge/booster word lists |
+| Results (CSV) | `reports/` | 161 | ~4.5MB | Head rankings, layer sweeps, summaries |
+| Results (JSON) | `reports/` | 137 | ~2.8MB | Metrics, analyses, manifests |
+| Paper Supplements | `paper/supplement/` | 20+ | ~150K | Bootstrap CI, calibration, validation data |
+
+**Total data to track with DVC: ~7.4 MB across 355+ files**
+
+### Data Already Gitignored (stays ignored)
+
+- `lab/runs/*` - Empty (only .gitkeep)
+- `mlruns/*` - Empty (only .gitkeep)
+- `*.png`, `*.html`, `*.pdf` - Generated plots
+- `*.log` - Generated logs
+- `*.ipynb` - Jupyter notebooks
+
+### Code/Config (stays in Git)
+
+- `lab/configs/*.json` - 100 experiment configs (~124K)
+- All Python scripts (analysis, training, figures)
+- LaTeX source files
+- Documentation and makefiles
+
+---
+
+## Proposed Directory Structure
+
+### Option A: Minimal Restructure (RECOMMENDED)
+
+Keep existing paths but organize DVC tracking by purpose. Minimal code changes required.
+
+```
+tinyLab/
+├── .dvc/                      # DVC configuration
+├── .dvcstore/                 # Local DVC cache (gitignored)
+│
+├── data/                      # Raw data - DVC tracked
+│   ├── lexicons/              # [DVC] Lexicon files
+│   │   └── hedge_booster.json
+│   └── README.md              # Documents data sources
+│
+├── lab/
+│   ├── data/                  # Lab datasets - DVC tracked
+│   │   ├── corpora/           # [DVC] Raw experimental corpora
+│   │   │   ├── facts_*.jsonl
+│   │   │   ├── counterfactual_*.jsonl
+│   │   │   ├── logical_*.jsonl
+│   │   │   └── negation_*.jsonl
+│   │   └── splits/            # [DVC] Processed train/test splits
+│   │       └── *.split.json
+│   │
+│   ├── configs/               # [GIT] Experiment configurations
+│   ├── analysis/              # [GIT] Analysis scripts
+│   ├── runs/                  # [IGNORED] Generated training runs
+│   └── tests/                 # [GIT] Test files
+│
+├── reports/                   # Results - DVC tracked
+│   ├── *.csv                  # [DVC] All ranking CSVs
+│   ├── *.json                 # [DVC] All metric JSONs
+│   ├── layer_sweep_*/         # [DVC] Layer sweep subdirs
+│   ├── appendices/            # [DVC] Additional analyses
+│   ├── RESULTS_MANIFEST.json  # [DVC] Master results index
+│   └── README.md              # Documents results structure
+│
+├── paper/
+│   ├── sections/              # [GIT] LaTeX source
+│   ├── scripts/               # [GIT] Figure generation scripts
+│   ├── supplement/            # Paper supplement data - DVC tracked
+│   │   ├── *.json             # [DVC] Supplement metrics
+│   │   └── cuda_validation/   # [DVC] CUDA validation results
+│   └── generated/             # [IGNORED] Auto-generated content
+│
+├── mlruns/                    # [IGNORED] MLflow tracking
+├── figs/                      # [GIT] Figure descriptions (md)
+│                              # [IGNORED] Rendered plots (png/pdf)
+├── docs/                      # [GIT] Documentation
+└── devlog/                    # [GIT] Development logs
+```
+
+**DVC Tracking Scheme:**
+- `data/lexicons/*.json` → Track individual files
+- `lab/data/corpora/` → Track entire directory
+- `lab/data/splits/` → Track entire directory
+- `reports/` → Track entire directory (includes all CSV/JSON)
+- `paper/supplement/` → Track entire directory
+
+---
+
+### Option B: Full Reorganization (More disruptive)
+
+Complete restructure following canonical data science layout. Requires updating all import paths.
+
+```
+tinyLab/
+├── .dvc/
+├── .dvcstore/                 # Local DVC cache
+│
+├── data/                      # ALL data - DVC tracked
+│   ├── raw/                   # Raw immutable data
+│   │   ├── corpora/           # [DVC] Moved from lab/data/corpora/
+│   │   │   ├── facts/
+│   │   │   ├── counterfactual/
+│   │   │   ├── logical/
+│   │   │   └── negation/
+│   │   └── lexicons/          # [DVC] Moved from data/lexicons/
+│   │
+│   └── processed/             # Derived/transformed data
+│       └── splits/            # [DVC] Moved from lab/data/splits/
+│
+├── results/                   # Renamed from reports/ - DVC tracked
+│   ├── metrics/               # [DVC] JSON metric files
+│   ├── rankings/              # [DVC] CSV ranking files
+│   ├── analyses/              # [DVC] Specialized analyses
+│   │   ├── layer_sweep/
+│   │   ├── vdi_drift/
+│   │   ├── entropy/
+│   │   └── ov_reports/
+│   └── MANIFEST.json          # Master index
+│
+├── models/                    # For future model artifacts
+│   └── checkpoints/           # [DVC] Training checkpoints (currently empty)
+│
+├── lab/                       # Experimental code
+│   ├── configs/               # [GIT] Experiment configs
+│   ├── analysis/              # [GIT] Analysis scripts
+│   └── tests/                 # [GIT] Test files
+│
+├── paper/
+│   ├── supplement/            # [DVC] Paper supplement data
+│   └── sections/              # [GIT] LaTeX source
+│
+├── logs/                      # Execution logs
+│   ├── mlruns/                # [IGNORED] MLflow runs
+│   └── training/              # [IGNORED] Training logs
+│
+└── notebooks/                 # [IGNORED] Jupyter notebooks
+```
+
+**Migration Impact:**
+- Requires updating ~30 analysis scripts
+- Need to update Makefile paths
+- Configuration files need path updates
+- More maintenance but cleaner long-term
+
+---
+
+## Recommendation: Option A (Minimal Restructure)
+
+**Rationale:**
+1. **Low risk** - existing code continues to work
+2. **Fast migration** - can be completed in hours, not days
+3. **Reversible** - easy to rollback if needed
+4. **Sufficient** - achieves all DVC goals without unnecessary complexity
+
+The current structure is already reasonably well-organized:
+- `lab/data/` clearly separates experimental data
+- `reports/` is an established convention
+- `paper/supplement/` is logically placed
+
+We can achieve clean DVC tracking without restructuring.
+
+---
+
+## DVC Configuration
+
+### DVC Remote Structure
+
+```bash
+# Local remote inside repository (git-ignored)
+.dvcstore/
+  ├── files/
+  │   └── md5/              # Content-addressable storage
+  │       ├── ab/
+  │       │   └── cdef123...
+  │       └── ...
+  └── tmp/
+```
+
+**Configuration:**
+```bash
+# .dvc/config.local
+[core]
+    remote = localstore
+
+[remote "localstore"]
+    url = .dvcstore
+```
+
+**Future S3 Migration:**
+```bash
+# Just add remote and push
+dvc remote add s3store s3://tinylab-data/
+dvc remote default s3store
+dvc push
+```
+
+### .gitignore Updates
+
+Add to `.gitignore`:
+```gitignore
+# DVC
+/reports/*.csv
+/reports/*.json
+/reports/layer_sweep_*
+/reports/appendices
+/lab/data/corpora
+/lab/data/splits
+/data/lexicons
+/paper/supplement/*.json
+/paper/supplement/*.csv
+/paper/supplement/cuda_validation
+.dvcstore/
+```
+
+Keep tracking:
+- `*.dvc` files (DVC pointers)
+- `.dvc/config` (DVC configuration)
+- `.dvc/.gitignore`
+
+---
+
+## DVC Tracking Strategy
+
+### Granularity Decision Matrix
+
+| Directory | Strategy | Rationale |
+|-----------|----------|-----------|
+| `lab/data/corpora/` | Single `.dvc` for entire dir | Files change together, versioned as unit |
+| `lab/data/splits/` | Single `.dvc` for entire dir | Derived from corpora, versioned together |
+| `data/lexicons/` | Individual `.dvc` per file | Small, independent files |
+| `reports/` | Single `.dvc` for entire dir | Results regenerated together, large file count |
+| `paper/supplement/` | Single `.dvc` for entire dir | Small, versioned with paper |
+
+### Directory-Level Tracking
+
+```bash
+# Track entire directories
+dvc add lab/data/corpora
+dvc add lab/data/splits
+dvc add reports
+dvc add paper/supplement
+
+# Track individual files
+dvc add data/lexicons/hedge_booster.json
+```
+
+**Generated artifacts:**
+```
+lab/data/corpora.dvc          # Pointer file (goes in git)
+lab/data/splits.dvc           # Pointer file (goes in git)
+reports.dvc                   # Pointer file (goes in git)
+paper/supplement.dvc          # Pointer file (goes in git)
+data/lexicons/hedge_booster.json.dvc  # Pointer file (goes in git)
+```
+
+---
+
+## Migration Workflow
+
+### Phase 1: Preparation (No changes to working tree)
+
+1. Create branch: `git checkout -b dvc-migration`
+2. Install DVC: `pip install dvc`
+3. Initialize DVC: `dvc init`
+4. Configure local remote:
+   ```bash
+   dvc remote add localstore .dvcstore --local
+   dvc remote default localstore
+   ```
+5. Update `.gitignore` with DVC patterns
+
+### Phase 2: Add DVC Tracking
+
+**Track data directories:**
+```bash
+# Add DVC tracking (data moved to .dvcstore, .dvc pointers created)
+dvc add lab/data/corpora
+dvc add lab/data/splits
+dvc add data/lexicons/hedge_booster.json
+dvc add reports
+dvc add paper/supplement
+
+# Check what was created
+ls -la lab/data/*.dvc
+ls -la *.dvc
+ls -la paper/*.dvc
+```
+
+**Commit DVC pointers:**
+```bash
+git add lab/data/corpora.dvc lab/data/splits.dvc
+git add data/lexicons/hedge_booster.json.dvc
+git add reports.dvc paper/supplement.dvc
+git add .gitignore .dvc/config .dvc/.gitignore
+git commit -m "Add DVC tracking for datasets, results, and supplements"
+```
+
+### Phase 3: Verification
+
+**Test data retrieval:**
+```bash
+# Remove data (simulate fresh clone)
+rm -rf lab/data/corpora lab/data/splits reports paper/supplement
+rm -f data/lexicons/hedge_booster.json
+
+# Restore from DVC
+dvc pull
+
+# Verify all files restored
+ls lab/data/corpora/*.jsonl
+ls lab/data/splits/*.json
+ls reports/*.csv
+ls paper/supplement/*.json
+```
+
+**Test reproducibility:**
+```bash
+# Run smoke test
+python smoke_test.py
+
+# Run single analysis
+python lab/analysis/export_head_rankings.py
+
+# Verify outputs match
+```
+
+### Phase 4: Documentation and Push
+
+```bash
+# Create comprehensive docs
+# (see Documentation section below)
+
+# Push to remote
+git push -u origin dvc-migration
+
+# Create pull request for review
+```
+
+---
+
+## Data Flows and Dependencies
+
+### Data Generation Pipeline
+
+```
+Raw Data (DVC)
+  ↓
+lab/data/corpora/*.jsonl
+  ↓
+[scripts/facts_make_split.py]
+  ↓
+lab/data/splits/*.json (DVC)
+  ↓
+[lab/analysis/*.py scripts]
+  ↓
+reports/*.csv + *.json (DVC)
+  ↓
+[paper/scripts/*.py]
+  ↓
+paper/supplement/*.json (DVC)
+  ↓
+[pdflatex]
+  ↓
+paper/main.pdf (IGNORED)
+```
+
+### Reproducibility Requirements
+
+To regenerate all results from scratch:
+
+```bash
+# 1. Clone repository
+git clone <repo> && cd tinyLab
+
+# 2. Restore data
+dvc pull
+
+# 3. Install dependencies
+pip install -e .
+
+# 4. Run analyses
+make postprocess
+
+# 5. Generate paper
+cd paper && make
+```
+
+**Critical insight:** Only raw data and splits need DVC tracking. Results can be regenerated via `make postprocess`, but we track them anyway for:
+- **Speed** - Avoid re-running expensive analyses
+- **Reproducibility** - Preserve exact results for papers
+- **Collaboration** - Share results without re-computation
+
+---
+
+## Documentation Requirements
+
+### 1. DVC_SETUP.md (New file)
+
+```markdown
+# DVC Setup Guide for tinyLab
+
+## Installation
+
+# Prerequisites
+- Python 3.11+
+- Git
+
+# Install DVC
+pip install dvc
+
+## First-time Setup (after cloning)
+
+# Pull all data
+dvc pull
+
+# Verify
+ls lab/data/corpora/*.jsonl
+ls reports/*.csv
+
+## Adding New Data
+
+# Track new dataset
+dvc add data/new_dataset.csv
+git add data/new_dataset.csv.dvc
+git commit -m "Add new dataset"
+
+## Updating Tracked Data
+
+# Modify data, then update tracking
+dvc add reports/
+git add reports.dvc
+git commit -m "Update results after experiment X"
+
+## Troubleshooting
+
+See docs/DVC_TROUBLESHOOTING.md
+```
+
+### 2. Update README.md
+
+Add DVC section:
+```markdown
+## Data Management with DVC
+
+This project uses DVC to manage datasets and results. After cloning:
+
+\`\`\`bash
+pip install dvc
+dvc pull
+\`\`\`
+
+See [DVC_SETUP.md](DVC_SETUP.md) for detailed instructions.
+```
+
+### 3. Update REPLICATION.md
+
+Add DVC step:
+```markdown
+## Replication Steps
+
+1. Clone repository
+2. **Pull data with DVC**: `dvc pull`
+3. Install dependencies: `pip install -e .`
+4. Run experiments: `make postprocess`
+```
+
+---
+
+## Migration Risks and Mitigations
+
+### Risk 1: Large file count in single .dvc file
+
+**Issue:** `reports/` has 298 files. If any single file changes, entire directory re-uploads.
+
+**Mitigation:**
+- Acceptable for ~7MB total size
+- Can split later if needed: `reports/csv.dvc` + `reports/json.dvc`
+- Monitor with `dvc status`
+
+### Risk 2: Git repository growth
+
+**Issue:** Multiple versions of `.dvc` files increase git repo size.
+
+**Mitigation:**
+- `.dvc` files are tiny (~100 bytes each)
+- Only 5 `.dvc` files total
+- Git handles small text files efficiently
+
+### Risk 3: Accidental data loss
+
+**Issue:** `dvc add` moves data to `.dvcstore`, could lose if .dvcstore deleted.
+
+**Mitigation:**
+- Create backup before migration: `tar czf tinylab-backup.tar.gz reports/ lab/data/`
+- Test `dvc pull` restoration before deleting original data
+- Keep branch protection on main/master
+
+### Risk 4: Path breakage
+
+**Issue:** Scripts might hardcode paths that DVC changes.
+
+**Mitigation:**
+- Option A (recommended) doesn't change any paths
+- DVC creates symlinks/copies, paths remain valid
+- Test suite runs before/after migration
+
+### Risk 5: Merge conflicts with .dvc files
+
+**Issue:** Two branches updating same data creates conflicts in `.dvc` files.
+
+**Mitigation:**
+- `.dvc` files are structured JSON, easy to merge
+- Use `dvc diff` to understand changes
+- Document conflict resolution in DVC_SETUP.md
+
+---
+
+## Testing Checklist
+
+Before considering migration complete:
+
+- [ ] `dvc status` shows all files tracked
+- [ ] `dvc push` succeeds to localstore
+- [ ] `dvc pull` restores all files correctly
+- [ ] `python smoke_test.py` passes
+- [ ] `make postprocess` completes without errors
+- [ ] `cd paper && make` generates PDF
+- [ ] All analysis scripts run successfully
+- [ ] Git repository size reasonable (<50MB)
+- [ ] `.dvcstore` size matches expected (~7-8MB)
+- [ ] Fresh clone + `dvc pull` works on different machine
+- [ ] Documentation clear and complete
+
+---
+
+## Future Enhancements
+
+### Phase 2: Cloud Storage (S3/GCS)
+
+```bash
+# Add S3 remote
+dvc remote add s3store s3://tinylab-data/dvc-cache
+dvc remote default s3store
+
+# Push to S3
+dvc push
+
+# Configure access
+dvc remote modify s3store access_key_id XXX
+dvc remote modify s3store secret_access_key YYY
+```
+
+### Phase 3: Data Versioning
+
+```bash
+# Tag dataset versions
+git tag -a data-v1.0 -m "Initial dataset release"
+git tag -a data-v1.1 -m "Added balanced variants"
+
+# Checkout specific version
+git checkout data-v1.0
+dvc checkout
+```
+
+### Phase 4: Pipelines (Optional)
+
+Define data pipelines in `dvc.yaml`:
+```yaml
+stages:
+  split_data:
+    cmd: python scripts/facts_make_split.py
+    deps:
+      - lab/data/corpora/
+    outs:
+      - lab/data/splits/
+
+  analyze:
+    cmd: python lab/analysis/head_rank_stats.py
+    deps:
+      - lab/data/splits/
+    outs:
+      - reports/h1_head_rank_stats.json
+```
+
+Run with: `dvc repro`
+
+---
+
+## Appendix: File Size Analysis
+
+### Files by Size Category
+
+| Size Range | Count | Category | DVC Strategy |
+|------------|-------|----------|--------------|
+| < 1KB | 45 | Config JSON, small JSONs | Track individually or as dir |
+| 1-10KB | 89 | Data splits, small metrics | Track as directory |
+| 10-50KB | 156 | Corpora, CSVs, metric JSONs | Track as directory |
+| 50-100KB | 48 | Large CSVs, result manifests | Track as directory |
+| 100KB-1MB | 15 | Large result files | Track as directory |
+| > 1MB | 2 | Comprehensive reports | Track as directory |
+
+**Total:** 355 files, ~7.4 MB
+
+### Growth Projections
+
+**Conservative (1 year):**
+- New experiments: 10 runs/month × 12 months = 120 runs
+- New results: ~200KB per run = 24MB
+- New checkpoints: 0 (using pretrained models)
+- **Total:** ~31MB
+
+**Aggressive (1 year):**
+- New experiments: 50 runs/month × 12 months = 600 runs
+- New results: ~200KB per run = 120MB
+- Model fine-tuning: 5 checkpoints × 500MB = 2.5GB
+- **Total:** ~2.6GB
+
+**Conclusion:** Even aggressive growth is manageable with S3/GCS backends.
+
+---
+
+## Appendix: DVC Commands Reference
+
+### Essential Commands
+
+```bash
+# Initialize
+dvc init
+
+# Track data
+dvc add <path>
+
+# Save changes
+git add <path>.dvc .gitignore
+git commit -m "Track <path> with DVC"
+
+# Push/pull data
+dvc push                    # Upload to remote
+dvc pull                    # Download from remote
+
+# Status
+dvc status                  # Check for changes
+dvc diff                    # Compare versions
+
+# Restore data
+dvc checkout                # Restore to committed version
+dvc fetch                   # Download without checking out
+```
+
+### Advanced Commands
+
+```bash
+# Remote management
+dvc remote add <name> <url>
+dvc remote modify <name> <option> <value>
+dvc remote list
+
+# Data management
+dvc gc -w                   # Clean up unused cache
+dvc cache dir               # Show cache location
+
+# Versioning
+dvc get <repo> <path>       # Download specific file
+dvc import <repo> <path>    # Import and track from another repo
+```
+
+---
+
+## Questions for Team Review
+
+Before proceeding with implementation, please confirm:
+
+1. **Structure:** Option A (minimal) or Option B (full reorganization)?
+2. **Granularity:** Single `reports.dvc` or split by type (`reports/csv.dvc`, `reports/json.dvc`)?
+3. **Configs:** Should `lab/configs/*.json` move to DVC? (Currently recommended: stay in git)
+4. **Generated outputs:** Confirm `*.png`, `*.pdf`, `*.html` should remain gitignored?
+5. **Timeline:** Migration in single PR or phased approach?
+
+---
+
+**Document Version:** 1.0
+**Date:** 2025-11-18
+**Author:** Claude
+**Status:** Draft for Review
diff --git a/DVC_SETUP.md b/DVC_SETUP.md
new file mode 100644
index 0000000..518f375
--- /dev/null
+++ b/DVC_SETUP.md
@@ -0,0 +1,609 @@
+# DVC Setup Guide for tinyLab
+
+This guide covers setting up and using DVC (Data Version Control) in the tinyLab project.
+
+## Table of Contents
+
+1. [Quick Start](#quick-start)
+2. [Installation](#installation)
+3. [First-Time Setup](#first-time-setup)
+4. [Daily Workflows](#daily-workflows)
+5. [Adding New Data](#adding-new-data)
+6. [Updating Tracked Data](#updating-tracked-data)
+7. [Troubleshooting](#troubleshooting)
+8. [Advanced Usage](#advanced-usage)
+
+---
+
+## Quick Start
+
+For new contributors cloning the repository:
+
+```bash
+# 1. Clone the repository
+git clone <repository-url>
+cd tinyLab
+
+# 2. Install DVC
+pip install dvc
+
+# 3. Pull all data
+dvc pull
+
+# 4. Verify data
+ls lab/data/corpora/*.jsonl
+ls reports/*.csv
+
+# 5. Run tests
+python smoke_test.py
+```
+
+That's it! You now have all datasets and results.
+
+---
+
+## Installation
+
+### Prerequisites
+
+- Python 3.11 or higher
+- Git
+- pip or conda
+
+### Install DVC
+
+**Using pip:**
+```bash
+pip install dvc
+```
+
+**Using conda:**
+```bash
+conda install -c conda-forge dvc
+```
+
+**Verify installation:**
+```bash
+dvc version
+# Should output: 3.x.x or higher
+```
+
+### Optional: Install cloud storage support
+
+If you'll be working with S3, GCS, or Azure later:
+
+```bash
+# For S3
+pip install 'dvc[s3]'
+
+# For Google Cloud Storage
+pip install 'dvc[gs]'
+
+# For Azure
+pip install 'dvc[azure]'
+
+# For all backends
+pip install 'dvc[all]'
+```
+
+---
+
+## First-Time Setup
+
+### After Cloning the Repository
+
+1. **Navigate to the repository:**
+   ```bash
+   cd tinyLab
+   ```
+
+2. **Install DVC** (if not already installed):
+   ```bash
+   pip install dvc
+   ```
+
+3. **Pull all tracked data:**
+   ```bash
+   dvc pull
+   ```
+
+   This downloads:
+   - Raw datasets (`lab/data/corpora/`)
+   - Data splits (`lab/data/splits/`)
+   - Lexicons (`data/lexicons/`)
+   - Results and metrics (`reports/`)
+   - Paper supplements (`paper/supplement/`)
+
+4. **Verify data integrity:**
+   ```bash
+   # Check DVC status
+   dvc status
+
+   # Should output: "Data and pipelines are up to date."
+
+   # Verify files exist
+   ls -lh lab/data/corpora/
+   ls -lh reports/
+   ```
+
+5. **Run smoke test:**
+   ```bash
+   python smoke_test.py
+   ```
+
+   If this passes, your setup is complete!
+
+---
+
+## Daily Workflows
+
+### Checking Data Status
+
+```bash
+# Check if your data is up to date
+dvc status
+
+# See what changed
+dvc diff
+
+# Compare with a specific commit
+dvc diff HEAD~1
+```
+
+### Pulling Latest Data
+
+When collaborators update datasets or results:
+
+```bash
+# Update code
+git pull
+
+# Update data
+dvc pull
+```
+
+**Pro tip:** Create a git alias:
+```bash
+git config alias.dvc-pull '!git pull && dvc pull'
+
+# Now you can use:
+git dvc-pull
+```
+
+### Updating to a Specific Version
+
+```bash
+# Checkout a specific commit
+git checkout <commit-hash>
+
+# Update data to match that commit
+dvc checkout
+
+# Return to latest
+git checkout main
+dvc checkout
+```
+
+---
+
+## Adding New Data
+
+### Adding a New Dataset
+
+1. **Place your data in the appropriate directory:**
+   ```bash
+   # Example: new corpus file
+   cp new_data.jsonl lab/data/corpora/
+   ```
+
+2. **Update DVC tracking:**
+   ```bash
+   # If the directory is already tracked, just update it
+   dvc add lab/data/corpora
+
+   # If it's a new standalone file
+   dvc add data/new_dataset.csv
+   ```
+
+3. **Commit the changes:**
+   ```bash
+   # Add the updated .dvc pointer file
+   git add lab/data/corpora.dvc
+
+   # Commit
+   git commit -m "Add new corpus data: new_data.jsonl"
+   ```
+
+4. **Push data to remote:**
+   ```bash
+   dvc push
+   ```
+
+### Adding a New Results Directory
+
+```bash
+# Add new results
+dvc add reports/new_experiment/
+
+# Commit
+git add reports/new_experiment.dvc .gitignore
+git commit -m "Add results from new experiment"
+
+# Push
+dvc push
+```
+
+---
+
+## Updating Tracked Data
+
+### Updating Existing Data
+
+When you regenerate results or modify datasets:
+
+1. **Make your changes** (run analysis, update data, etc.)
+
+2. **Update DVC tracking:**
+   ```bash
+   # Update directory tracking
+   dvc add reports/
+
+   # DVC will detect changes and update the .dvc file
+   ```
+
+3. **Commit the updated pointer:**
+   ```bash
+   git add reports.dvc
+   git commit -m "Update results after fixing analysis bug"
+   ```
+
+4. **Push to remote:**
+   ```bash
+   dvc push
+   ```
+
+### Example: Regenerating All Results
+
+```bash
+# Regenerate results
+make postprocess
+
+# Update DVC tracking
+dvc add reports/
+dvc add paper/supplement/
+
+# Commit
+git add reports.dvc paper/supplement.dvc
+git commit -m "Regenerate results with updated analysis scripts"
+
+# Push
+dvc push
+```
+
+---
+
+## Troubleshooting
+
+### Issue: `dvc pull` fails with "file not found"
+
+**Cause:** Data not pushed to remote, or remote not configured.
+
+**Solution:**
+```bash
+# Check remote configuration
+dvc remote list
+
+# Should show:
+# localstore    .dvcstore
+
+# If empty, reconfigure
+dvc remote add localstore .dvcstore --local
+dvc remote default localstore
+```
+
+### Issue: Large data in git repository
+
+**Cause:** Accidentally committed data files instead of .dvc pointers.
+
+**Solution:**
+```bash
+# Remove data from git, keep locally
+git rm --cached reports/*.csv
+git rm --cached reports/*.json
+
+# Track with DVC
+dvc add reports/
+
+# Commit
+git add reports.dvc .gitignore
+git commit -m "Fix: Move reports to DVC tracking"
+```
+
+### Issue: "Data is not in cache"
+
+**Cause:** `.dvcstore` was deleted or data never pushed.
+
+**Solution:**
+```bash
+# If you have the data files locally
+dvc add <directory>
+dvc push
+
+# If not, pull from remote
+dvc pull
+
+# If all else fails, regenerate
+make postprocess
+dvc add reports/
+git add reports.dvc
+git commit -m "Regenerate missing results"
+```
+
+### Issue: Slow `dvc pull`
+
+**Cause:** Many files, slow I/O, or network issues.
+
+**Solutions:**
+```bash
+# Pull only specific directory
+dvc pull reports.dvc
+
+# Use parallel jobs
+dvc pull -j 4
+
+# Check cache location (move to faster disk if needed)
+dvc cache dir
+```
+
+### Issue: Merge conflict in .dvc file
+
+**Example conflict in `reports.dvc`:**
+```diff
+<<<<<<< HEAD
+- md5: abc123
+=======
+- md5: def456
+>>>>>>> feature-branch
+```
+
+**Solution:**
+```bash
+# Keep one version (choose based on which data you want)
+# Edit reports.dvc to resolve conflict
+
+# Restore data matching the chosen MD5
+dvc checkout reports.dvc
+
+# Complete the merge
+git add reports.dvc
+git commit
+```
+
+For more details, see [DVC_TROUBLESHOOTING.md](DVC_TROUBLESHOOTING.md).
+
+---
+
+## Advanced Usage
+
+### Working with Multiple Remotes
+
+```bash
+# Add S3 remote
+dvc remote add s3store s3://tinylab-data/dvc-cache
+dvc remote modify s3store region us-west-2
+
+# Add GCS remote
+dvc remote add gcsstore gs://tinylab-data/dvc-cache
+
+# Push to specific remote
+dvc push -r s3store
+
+# Set default
+dvc remote default s3store
+```
+
+### Configuring AWS Credentials
+
+```bash
+# Option 1: Environment variables (recommended)
+export AWS_ACCESS_KEY_ID=<your-key>
+export AWS_SECRET_ACCESS_KEY=<your-secret>
+
+# Option 2: DVC config (not recommended for credentials)
+dvc remote modify s3store access_key_id <your-key>
+dvc remote modify s3store secret_access_key <your-secret>
+
+# Option 3: Use AWS CLI credentials
+# DVC will automatically use ~/.aws/credentials
+```
+
+### Data Registry (Sharing Across Projects)
+
+```bash
+# Import data from another DVC project
+dvc import https://github.com/org/ml-data data/raw/dataset.csv
+
+# This tracks the external source and can be updated
+dvc update dataset.csv.dvc
+```
+
+### DVC Pipelines (Future)
+
+For reproducible workflows, define stages in `dvc.yaml`:
+
+```yaml
+stages:
+  preprocess:
+    cmd: python scripts/facts_make_split.py
+    deps:
+      - lab/data/corpora/
+      - scripts/facts_make_split.py
+    outs:
+      - lab/data/splits/
+
+  analyze:
+    cmd: python lab/analysis/export_head_rankings.py
+    deps:
+      - lab/data/splits/
+      - lab/analysis/export_head_rankings.py
+    outs:
+      - reports/
+```
+
+Then run with:
+```bash
+dvc repro
+```
+
+### Garbage Collection
+
+Clean up old unused data:
+
+```bash
+# Dry run - see what would be removed
+dvc gc -w -vv
+
+# Actually remove
+dvc gc -w
+
+# Keep only data for last 3 commits
+dvc gc -w --all-commits --rev HEAD~3
+```
+
+### Metrics and Plots
+
+Track metrics for experiment comparison:
+
+```bash
+# Track metrics file
+dvc metrics show reports/metrics.json
+
+# Compare metrics across branches
+dvc metrics diff main feature-branch
+
+# Show plots
+dvc plots show reports/training_curve.csv
+```
+
+---
+
+## Integration with Git Workflows
+
+### Feature Branch Workflow
+
+```bash
+# Create feature branch
+git checkout -b feature/new-experiment
+
+# Run experiment, generate new results
+python run_experiment.py
+
+# Track results
+dvc add reports/
+git add reports.dvc
+git commit -m "Add results for new experiment"
+
+# Push code and data
+git push origin feature/new-experiment
+dvc push
+
+# Create PR
+gh pr create
+```
+
+### Reviewing PRs with Data Changes
+
+As a reviewer:
+
+```bash
+# Checkout PR branch
+gh pr checkout 123
+
+# Pull data changes
+dvc pull
+
+# Review data
+ls reports/
+head reports/new_results.csv
+
+# Compare with main
+dvc diff main
+```
+
+---
+
+## Best Practices
+
+### DO:
+- ✅ Always run `dvc pull` after `git pull`
+- ✅ Use `dvc status` before committing to check for data changes
+- ✅ Add descriptive commit messages when updating data
+- ✅ Push data (`dvc push`) after pushing code (`git push`)
+- ✅ Use `.dvcignore` for files that shouldn't be tracked
+
+### DON'T:
+- ❌ Commit large data files directly to git
+- ❌ Delete `.dvcstore` without backing up
+- ❌ Modify `.dvc` files manually (use `dvc add` instead)
+- ❌ Push to git without pushing to DVC (others can't get data)
+- ❌ Use `git add .` blindly (might add large files)
+
+---
+
+## Quick Reference
+
+### Common Commands
+
+| Command | Purpose |
+|---------|---------|
+| `dvc pull` | Download data from remote |
+| `dvc push` | Upload data to remote |
+| `dvc status` | Check if data is up to date |
+| `dvc diff` | Show data changes |
+| `dvc add <path>` | Track file/directory with DVC |
+| `dvc checkout` | Restore data to match git HEAD |
+| `dvc remote list` | Show configured remotes |
+| `dvc cache dir` | Show cache location |
+
+### Directory Structure
+
+```
+tinyLab/
+├── .dvc/                   # DVC configuration
+│   ├── config              # Remote settings (tracked in git)
+│   ├── config.local        # Local settings (not tracked)
+│   └── .gitignore          # What to ignore in .dvc/
+│
+├── .dvcstore/              # Local data cache (not tracked in git)
+│
+├── lab/data/
+│   ├── corpora/            # [DVC tracked] Raw datasets
+│   ├── corpora.dvc         # [Git tracked] Pointer to data
+│   └── splits.dvc          # [Git tracked] Pointer to splits
+│
+└── reports/                # [DVC tracked] Results
+    └── reports.dvc         # [Git tracked] Pointer to results
+```
+
+---
+
+## Getting Help
+
+- **DVC Documentation:** https://dvc.org/doc
+- **DVC Discord:** https://dvc.org/chat
+- **Project Issues:** See GitHub issues
+- **Internal Docs:** See [DVC_TROUBLESHOOTING.md](DVC_TROUBLESHOOTING.md)
+
+---
+
+## Next Steps
+
+- Read [DVC_MIGRATION_DESIGN.md](DVC_MIGRATION_DESIGN.md) for architecture details
+- See [REPLICATION.md](docs/REPLICATION.md) for full reproduction instructions
+- Join the discussion in project Slack/Discord
+
+---
+
+**Last Updated:** 2025-11-18
+**DVC Version:** 3.x
+**Maintained By:** tinyLab Team
diff --git a/DVC_TROUBLESHOOTING.md b/DVC_TROUBLESHOOTING.md
new file mode 100644
index 0000000..d16aae0
--- /dev/null
+++ b/DVC_TROUBLESHOOTING.md
@@ -0,0 +1,799 @@
+# DVC Troubleshooting Guide
+
+Comprehensive troubleshooting for common DVC issues in tinyLab.
+
+## Table of Contents
+
+1. [Installation Issues](#installation-issues)
+2. [Data Retrieval Problems](#data-retrieval-problems)
+3. [Cache Issues](#cache-issues)
+4. [Remote Storage Issues](#remote-storage-issues)
+5. [Performance Issues](#performance-issues)
+6. [Merge Conflicts](#merge-conflicts)
+7. [Git Integration Issues](#git-integration-issues)
+8. [Emergency Procedures](#emergency-procedures)
+
+---
+
+## Installation Issues
+
+### Issue: DVC command not found
+
+**Symptoms:**
+```bash
+$ dvc --version
+bash: dvc: command not found
+```
+
+**Solutions:**
+
+1. **Install DVC:**
+   ```bash
+   pip install dvc
+   ```
+
+2. **Check if installed in different environment:**
+   ```bash
+   which python
+   python -m dvc --version
+   ```
+
+3. **Use full path:**
+   ```bash
+   python -m dvc pull
+   ```
+
+4. **Reinstall:**
+   ```bash
+   pip uninstall dvc
+   pip install --no-cache-dir dvc
+   ```
+
+### Issue: Import errors after installation
+
+**Symptoms:**
+```
+ImportError: cannot import name 'x' from 'dvc'
+```
+
+**Solutions:**
+
+1. **Check Python version:**
+   ```bash
+   python --version  # Should be 3.11+
+   ```
+
+2. **Reinstall with dependencies:**
+   ```bash
+   pip install --upgrade 'dvc[all]'
+   ```
+
+3. **Clear Python cache:**
+   ```bash
+   find . -type d -name __pycache__ -exec rm -rf {} +
+   pip install --force-reinstall dvc
+   ```
+
+---
+
+## Data Retrieval Problems
+
+### Issue: `dvc pull` fails with "file not found in cache"
+
+**Symptoms:**
+```
+ERROR: failed to pull data from the cloud - ... file not found
+```
+
+**Diagnosis:**
+```bash
+# Check remote configuration
+dvc remote list
+
+# Check what DVC is looking for
+dvc status -v
+```
+
+**Solutions:**
+
+1. **Data never pushed - regenerate locally:**
+   ```bash
+   # Regenerate all results
+   make postprocess
+
+   # Re-add to DVC
+   dvc add reports/
+   dvc push
+   ```
+
+2. **Remote misconfigured:**
+   ```bash
+   # Fix local remote
+   dvc remote add localstore .dvcstore --local
+   dvc remote default localstore
+
+   # Check if .dvcstore exists
+   ls -la .dvcstore/
+   ```
+
+3. **Cache corrupted - fetch from another source:**
+   ```bash
+   # If you have another clone with data
+   rsync -av other-clone/.dvcstore/ ./.dvcstore/
+
+   # Or pull from backup remote
+   dvc pull -r backup-remote
+   ```
+
+### Issue: `dvc pull` downloads nothing
+
+**Symptoms:**
+```bash
+$ dvc pull
+# No output, no data downloaded
+```
+
+**Diagnosis:**
+```bash
+dvc status
+# Should show: "Data and pipelines are up to date"
+# OR show missing files
+```
+
+**Solutions:**
+
+1. **Data already in cache:**
+   ```bash
+   # Restore from cache
+   dvc checkout
+   ```
+
+2. **No .dvc files in repository:**
+   ```bash
+   # Check for .dvc files
+   find . -name "*.dvc"
+
+   # If missing, you may need to:
+   git checkout main  # or correct branch
+   ```
+
+3. **Wrong branch:**
+   ```bash
+   git branch  # Check current branch
+   git checkout main
+   dvc pull
+   ```
+
+### Issue: Partial data downloaded
+
+**Symptoms:**
+Some directories restored, others missing.
+
+**Solutions:**
+
+1. **Pull specific targets:**
+   ```bash
+   # Pull each .dvc file explicitly
+   dvc pull lab/data/corpora.dvc
+   dvc pull lab/data/splits.dvc
+   dvc pull reports.dvc
+   dvc pull paper/supplement.dvc
+   ```
+
+2. **Force pull:**
+   ```bash
+   dvc pull --force
+   ```
+
+3. **Check for errors:**
+   ```bash
+   dvc pull -v  # Verbose mode
+   ```
+
+---
+
+## Cache Issues
+
+### Issue: Cache directory missing
+
+**Symptoms:**
+```
+ERROR: failed to pull data - cache directory not found
+```
+
+**Solutions:**
+
+1. **Restore cache from backup:**
+   ```bash
+   # If you have a backup
+   tar xzf tinylab_cache_backup.tar.gz -C .
+
+   # Verify
+   ls -la .dvcstore/
+   ```
+
+2. **Re-pull from remote:**
+   ```bash
+   dvc pull --force
+   ```
+
+3. **Regenerate data:**
+   ```bash
+   make postprocess
+   dvc add reports/
+   ```
+
+### Issue: Cache taking too much space
+
+**Symptoms:**
+`.dvcstore` directory is very large.
+
+**Diagnosis:**
+```bash
+# Check cache size
+du -sh .dvcstore
+
+# See what's in cache
+dvc cache dir
+ls -lh .dvcstore/files/md5/
+```
+
+**Solutions:**
+
+1. **Clean up old versions:**
+   ```bash
+   # Keep only data for current commit
+   dvc gc -w
+
+   # Keep only last 5 commits
+   dvc gc --all-commits --rev HEAD~5
+
+   # Dry run first
+   dvc gc -w -vv --dry
+   ```
+
+2. **Move cache to larger disk:**
+   ```bash
+   # Move cache
+   mv .dvcstore /mnt/large-disk/tinylab-dvc-cache
+
+   # Create symlink
+   ln -s /mnt/large-disk/tinylab-dvc-cache .dvcstore
+   ```
+
+3. **Use external cache:**
+   ```bash
+   # Configure shared cache location
+   dvc cache dir /shared/dvc-cache
+   dvc config cache.shared group
+   ```
+
+### Issue: Cache corrupted
+
+**Symptoms:**
+```
+ERROR: checksum mismatch
+ERROR: corrupted cache file
+```
+
+**Solutions:**
+
+1. **Remove corrupted file:**
+   ```bash
+   # DVC will tell you which file
+   # Example: .dvcstore/files/md5/ab/cdef123456
+   rm .dvcstore/files/md5/ab/cdef123456
+
+   # Re-pull
+   dvc pull --force
+   ```
+
+2. **Clear entire cache and re-pull:**
+   ```bash
+   # DANGER: Only if you have remote backup
+   rm -rf .dvcstore/*
+   dvc pull
+   ```
+
+3. **Verify integrity:**
+   ```bash
+   dvc status -v
+   dvc checkout --force
+   ```
+
+---
+
+## Remote Storage Issues
+
+### Issue: Cannot push to remote
+
+**Symptoms:**
+```
+ERROR: failed to push data to the cloud
+```
+
+**Diagnosis:**
+```bash
+# Check remote config
+dvc remote list
+dvc remote list --local
+
+# Test remote access
+dvc remote --help
+```
+
+**Solutions:**
+
+1. **Local remote - check path:**
+   ```bash
+   # Verify .dvcstore exists
+   mkdir -p .dvcstore
+
+   # Reconfigure
+   dvc remote add localstore .dvcstore --local --force
+   dvc remote default localstore
+   ```
+
+2. **S3 remote - check credentials:**
+   ```bash
+   # Check AWS credentials
+   aws s3 ls s3://your-bucket/
+
+   # Set credentials
+   export AWS_ACCESS_KEY_ID=xxx
+   export AWS_SECRET_ACCESS_KEY=yyy
+
+   # Or configure remote
+   dvc remote modify s3store access_key_id xxx
+   dvc remote modify s3store secret_access_key yyy
+   ```
+
+3. **Permission denied:**
+   ```bash
+   # For local remote
+   chmod -R u+w .dvcstore
+
+   # For S3/GCS - check IAM permissions
+   ```
+
+### Issue: Slow push/pull
+
+**Symptoms:**
+`dvc push` or `dvc pull` takes a very long time.
+
+**Solutions:**
+
+1. **Use parallel transfers:**
+   ```bash
+   dvc pull -j 8  # Use 8 parallel jobs
+   dvc push -j 8
+   ```
+
+2. **Check network:**
+   ```bash
+   # For S3
+   aws s3 ls s3://bucket/ --profile default
+
+   # Check region
+   dvc remote modify s3store region us-west-2
+   ```
+
+3. **Use faster remote:**
+   ```bash
+   # Add regional endpoint for S3
+   dvc remote modify s3store endpoint_url https://s3.us-west-2.amazonaws.com
+   ```
+
+4. **Compress large files:**
+   ```bash
+   # For text files, enable compression
+   gzip reports/*.csv
+   dvc add reports/
+   ```
+
+---
+
+## Performance Issues
+
+### Issue: `dvc status` is slow
+
+**Symptoms:**
+`dvc status` takes minutes to complete.
+
+**Solutions:**
+
+1. **Check for large directories:**
+   ```bash
+   # See what DVC is tracking
+   find . -name "*.dvc"
+
+   # If tracking too many files, consider splitting
+   ```
+
+2. **Use directory tracking instead of individual files:**
+   ```bash
+   # Instead of:
+   dvc add reports/*.csv  # Tracks each file separately
+
+   # Use:
+   dvc add reports/  # Tracks entire directory
+   ```
+
+3. **Disable status for CI:**
+   ```bash
+   # In CI/CD, skip status checks
+   dvc pull --no-status
+   ```
+
+### Issue: Git operations are slow
+
+**Symptoms:**
+`git status`, `git add` are slow after DVC migration.
+
+**Diagnosis:**
+```bash
+# Check git status
+time git status
+
+# Check if large files in git
+git ls-files | xargs du -sh | sort -h | tail -20
+```
+
+**Solutions:**
+
+1. **Remove large files from git:**
+   ```bash
+   # Find large files
+   git ls-files | xargs ls -lh | sort -k5 -h | tail -20
+
+   # Remove if shouldn't be in git
+   git rm --cached path/to/large/file
+   dvc add path/to/large/file
+   git add path/to/large/file.dvc
+   ```
+
+2. **Use git-lfs for PDFs/images:**
+   ```bash
+   # If you have large binary files that aren't tracked by DVC
+   git lfs track "*.pdf"
+   git lfs track "*.png"
+   ```
+
+3. **Check .gitignore:**
+   ```bash
+   # Ensure DVC data is ignored
+   cat .gitignore | grep dvc
+   ```
+
+---
+
+## Merge Conflicts
+
+### Issue: Conflict in .dvc file
+
+**Symptoms:**
+```diff
+<<<<<<< HEAD
+  md5: abc123456
+=======
+  md5: def789012
+>>>>>>> feature-branch
+```
+
+**Understanding:**
+- Both branches modified the same data
+- Git doesn't know which version to keep
+
+**Solutions:**
+
+**Option 1: Keep HEAD version**
+```bash
+# Edit .dvc file, remove conflict markers, keep HEAD md5
+vim reports.dvc  # or your editor
+
+# Restore data matching HEAD
+dvc checkout reports.dvc
+
+# Complete merge
+git add reports.dvc
+git commit
+```
+
+**Option 2: Keep incoming version**
+```bash
+# Edit .dvc file, keep incoming md5
+vim reports.dvc
+
+# Restore data matching incoming
+dvc checkout reports.dvc
+
+# Complete merge
+git add reports.dvc
+git commit
+```
+
+**Option 3: Regenerate data (recommended)**
+```bash
+# Merge code first
+git checkout HEAD reports.dvc  # or branch version
+git add reports.dvc
+git commit
+
+# Regenerate fresh data
+make postprocess
+
+# Re-track
+dvc add reports/
+git add reports.dvc
+git commit --amend -m "Merge with regenerated data"
+```
+
+### Issue: Both added .dvc file
+
+**Symptoms:**
+```
+CONFLICT (both added): reports.dvc
+```
+
+**Solution:**
+```bash
+# Choose one version
+git checkout --theirs reports.dvc
+# OR
+git checkout --ours reports.dvc
+
+# Verify data
+dvc checkout reports.dvc
+
+# If data doesn't match, regenerate
+make postprocess
+dvc add reports/
+git add reports.dvc
+git commit
+```
+
+---
+
+## Git Integration Issues
+
+### Issue: .dvc files not being tracked
+
+**Symptoms:**
+`git status` doesn't show `.dvc` files after `dvc add`.
+
+**Solutions:**
+
+1. **Explicitly add:**
+   ```bash
+   git add *.dvc
+   git add **/*.dvc
+   ```
+
+2. **Check .gitignore:**
+   ```bash
+   # Make sure .dvc files aren't ignored
+   grep "\.dvc$" .gitignore
+
+   # Should NOT have:
+   # *.dvc  # This would ignore pointer files
+   ```
+
+3. **Check DVC config:**
+   ```bash
+   cat .dvc/.gitignore
+   # Should NOT ignore *.dvc files
+   ```
+
+### Issue: Data files committed to git
+
+**Symptoms:**
+Large files tracked by both git and DVC.
+
+**Diagnosis:**
+```bash
+# Check file size in git
+git ls-files | xargs du -sh | sort -h | tail -20
+
+# Check if tracked by both
+git ls-files reports/*.csv
+ls -la reports/*.csv.dvc
+```
+
+**Solution:**
+```bash
+# Remove from git, keep locally
+git rm --cached reports/*.csv
+git rm --cached reports/*.json
+
+# Ensure tracked by DVC
+dvc status
+
+# Commit
+git add .gitignore
+git commit -m "Remove data files from git (tracked by DVC)"
+
+# Push
+git push
+```
+
+### Issue: Large git repository after DVC migration
+
+**Diagnosis:**
+```bash
+# Check repo size
+du -sh .git
+
+# Check large objects
+git rev-list --objects --all | \
+  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | \
+  awk '/^blob/ {print substr($0,6)}' | \
+  sort -k2 -n | tail -20
+```
+
+**Solution:**
+Use git-filter-repo to remove historical large files:
+```bash
+# Install git-filter-repo
+pip install git-filter-repo
+
+# Remove historical data files
+git filter-repo --path reports/ --invert-paths
+git filter-repo --path lab/data/corpora --invert-paths
+
+# Force push (DANGER: coordinate with team)
+git push --force
+```
+
+---
+
+## Emergency Procedures
+
+### Emergency: Lost all data
+
+**Situation:** `.dvcstore` deleted, no backup.
+
+**Recovery:**
+
+1. **Check remote:**
+   ```bash
+   dvc pull -v
+   ```
+
+2. **Check other clones:**
+   ```bash
+   # Copy from teammate's machine
+   rsync -av teammate-machine:/path/to/tinyLab/.dvcstore/ ./.dvcstore/
+   ```
+
+3. **Regenerate from scratch:**
+   ```bash
+   # Regenerate all results
+   make postprocess
+
+   # Re-track
+   dvc add reports/
+   dvc push
+
+   # Commit
+   git add reports.dvc
+   git commit -m "Regenerate lost results"
+   ```
+
+### Emergency: Corrupted DVC configuration
+
+**Situation:** DVC commands fail with config errors.
+
+**Recovery:**
+
+1. **Backup current config:**
+   ```bash
+   cp -r .dvc .dvc.backup
+   ```
+
+2. **Reinitialize:**
+   ```bash
+   rm -rf .dvc
+   dvc init
+
+   # Reconfigure remote
+   dvc remote add localstore .dvcstore --local
+   dvc remote default localstore
+   ```
+
+3. **Verify:**
+   ```bash
+   dvc status
+   dvc pull
+   ```
+
+### Emergency: Cannot push, about to lose data
+
+**Situation:** Need to save work but `dvc push` fails.
+
+**Workaround:**
+
+1. **Backup cache manually:**
+   ```bash
+   tar czf dvc-cache-backup-$(date +%Y%m%d).tar.gz .dvcstore/
+   ```
+
+2. **Push to alternative location:**
+   ```bash
+   # Temporary remote
+   dvc remote add backup /mnt/external-drive/tinylab-dvc --local
+   dvc push -r backup
+   ```
+
+3. **Upload to cloud manually:**
+   ```bash
+   # AWS S3
+   aws s3 sync .dvcstore/ s3://my-backup-bucket/tinylab-dvc-emergency/
+
+   # Later recover with:
+   aws s3 sync s3://my-backup-bucket/tinylab-dvc-emergency/ .dvcstore/
+   ```
+
+---
+
+## Diagnostic Commands
+
+Run these to gather information before asking for help:
+
+```bash
+# DVC version and config
+dvc version
+dvc config --list
+dvc config --list --local
+
+# Remote configuration
+dvc remote list
+dvc remote list --local
+
+# Status and cache
+dvc status -v
+dvc cache dir
+du -sh .dvcstore
+
+# Git integration
+git status
+find . -name "*.dvc"
+
+# System info
+df -h .
+python --version
+pip list | grep dvc
+```
+
+---
+
+## Getting More Help
+
+1. **Check DVC documentation:** https://dvc.org/doc/user-guide/troubleshooting
+2. **Search DVC forum:** https://discuss.dvc.org
+3. **File an issue:** Project GitHub issues
+4. **Ask in Slack/Discord:** See team channels
+
+When asking for help, include:
+- Output of diagnostic commands above
+- Full error message
+- What you were trying to do
+- What you expected to happen
+
+---
+
+## Prevention Tips
+
+Avoid issues by following these practices:
+
+1. **Always run `dvc status` before committing**
+2. **Test `dvc pull` on fresh clone regularly**
+3. **Keep backups of `.dvcstore` for critical projects**
+4. **Document custom remote configurations**
+5. **Use `--dry` flag for destructive operations**
+6. **Monitor `.git` and `.dvcstore` sizes**
+7. **Coordinate with team before force-pushing**
+
+---
+
+**Last Updated:** 2025-11-18
+**Maintained By:** tinyLab Team
diff --git a/README.md b/README.md
index 85249c1..6c291ec 100644
--- a/README.md
+++ b/README.md
@@ -55,6 +55,8 @@ python smoke_test_cuda.py  # optional sanity check
 - For CUDA-specific setup and optimization guide, see [docs/CUDA_SETUP.md](docs/CUDA_SETUP.md).
 - For an annotated walkthrough of the harness, see [QUICKSTART.md](QUICKSTART.md).
 
+### Stage‑1A Pilot Utilities
+
 To sanity‑check the new Stage‑1A utilities (Task‑B data, circularity metrics, VDI probe), run:
 
 ```bash
@@ -63,6 +65,41 @@ bash scripts/run_pilot_dry_run.sh
 
 This runs a small, end‑to‑end dry‑run on `gpt2-small` to validate the geometry and structural probes without training the 2‑layer pilot model.
 
+## Data Management with DVC
+
+This project uses [DVC (Data Version Control)](https://dvc.org) to manage datasets, results, and artifacts. DVC keeps large data files out of Git while maintaining full version control and reproducibility.
+
+### First-time Setup
+
+After cloning the repository, pull all tracked data:
+
+```bash
+# Install DVC
+pip install dvc
+
+# Pull datasets and results
+dvc pull
+```
+
+This downloads:
+- Raw datasets (`lab/data/corpora/`)
+- Data splits (`lab/data/splits/`)
+- Results and metrics (`reports/`)
+- Paper supplements (`paper/supplement/`)
+
+### Why DVC?
+
+- **Version control for data** - Track dataset and result versions alongside code
+- **Efficient storage** - Large files stored separately from Git
+- **Reproducibility** - Exact data versions tied to code commits
+- **Scalability** - Seamlessly migrate to S3/GCS/Azure when needed
+
+### Documentation
+
+- [DVC_SETUP.md](DVC_SETUP.md) - Complete setup and usage guide
+- [DVC_MIGRATION_DESIGN.md](DVC_MIGRATION_DESIGN.md) - Architecture and design decisions
+- [DVC_TROUBLESHOOTING.md](DVC_TROUBLESHOOTING.md) - Common issues and solutions
+
 ## Repository Layout
 
 | Path | Purpose |
diff --git a/scripts/migrate_to_dvc.sh b/scripts/migrate_to_dvc.sh
new file mode 100755
index 0000000..6b9bad4
--- /dev/null
+++ b/scripts/migrate_to_dvc.sh
@@ -0,0 +1,365 @@
+#!/usr/bin/env bash
+#
+# migrate_to_dvc.sh
+#
+# Automated migration script for adding DVC tracking to tinyLab
+#
+# Usage:
+#   ./scripts/migrate_to_dvc.sh [--dry-run] [--backup]
+#
+# Options:
+#   --dry-run    Show what would be done without making changes
+#   --backup     Create backup tarball before migration
+#   --help       Show this help message
+
+set -euo pipefail
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+BLUE='\033[0;34m'
+NC='\033[0m' # No Color
+
+# Configuration
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+BACKUP_DIR="${REPO_ROOT}/backups"
+TIMESTAMP=$(date +%Y%m%d_%H%M%S)
+
+# Flags
+DRY_RUN=false
+CREATE_BACKUP=false
+
+# Parse arguments
+while [[ $# -gt 0 ]]; do
+  case $1 in
+    --dry-run)
+      DRY_RUN=true
+      shift
+      ;;
+    --backup)
+      CREATE_BACKUP=true
+      shift
+      ;;
+    --help)
+      grep '^#' "$0" | sed 's/^# //' | sed 's/^#//'
+      exit 0
+      ;;
+    *)
+      echo -e "${RED}Unknown option: $1${NC}"
+      exit 1
+      ;;
+  esac
+done
+
+# Logging functions
+log_info() {
+  echo -e "${BLUE}[INFO]${NC} $1"
+}
+
+log_success() {
+  echo -e "${GREEN}[SUCCESS]${NC} $1"
+}
+
+log_warning() {
+  echo -e "${YELLOW}[WARNING]${NC} $1"
+}
+
+log_error() {
+  echo -e "${RED}[ERROR]${NC} $1"
+}
+
+# Dry run wrapper
+run_cmd() {
+  if [ "$DRY_RUN" = true ]; then
+    echo -e "${YELLOW}[DRY-RUN]${NC} Would run: $*"
+  else
+    "$@"
+  fi
+}
+
+# Check prerequisites
+check_prerequisites() {
+  log_info "Checking prerequisites..."
+
+  # Check if we're in the right directory
+  if [ ! -f "$REPO_ROOT/pyproject.toml" ] || [ ! -d "$REPO_ROOT/lab" ]; then
+    log_error "Must be run from tinyLab repository root"
+    exit 1
+  fi
+
+  # Check if git is available
+  if ! command -v git &> /dev/null; then
+    log_error "git is required but not installed"
+    exit 1
+  fi
+
+  # Check if DVC is installed
+  if ! command -v dvc &> /dev/null; then
+    log_error "DVC is not installed. Install with: pip install dvc"
+    exit 1
+  fi
+
+  # Check if we're on a clean branch
+  if [ -n "$(git status --porcelain)" ]; then
+    log_warning "Working directory has uncommitted changes"
+    if [ "$DRY_RUN" = false ]; then
+      read -p "Continue anyway? (y/N) " -n 1 -r
+      echo
+      if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+        exit 1
+      fi
+    fi
+  fi
+
+  log_success "Prerequisites check passed"
+}
+
+# Create backup
+create_backup() {
+  if [ "$CREATE_BACKUP" = true ]; then
+    log_info "Creating backup..."
+
+    mkdir -p "$BACKUP_DIR"
+    BACKUP_FILE="${BACKUP_DIR}/tinylab_pre_dvc_${TIMESTAMP}.tar.gz"
+
+    run_cmd tar czf "$BACKUP_FILE" \
+      lab/data/corpora \
+      lab/data/splits \
+      data/lexicons \
+      reports \
+      paper/supplement \
+      2>/dev/null || true
+
+    if [ -f "$BACKUP_FILE" ]; then
+      log_success "Backup created: $BACKUP_FILE"
+    fi
+  fi
+}
+
+# Initialize DVC
+init_dvc() {
+  log_info "Initializing DVC..."
+
+  cd "$REPO_ROOT"
+
+  # Check if already initialized
+  if [ -d ".dvc" ]; then
+    log_warning "DVC already initialized, skipping"
+    return 0
+  fi
+
+  run_cmd dvc init
+
+  log_success "DVC initialized"
+}
+
+# Configure local remote
+configure_remote() {
+  log_info "Configuring local DVC remote..."
+
+  cd "$REPO_ROOT"
+
+  # Add local remote
+  if ! dvc remote list | grep -q "localstore"; then
+    run_cmd dvc remote add localstore .dvcstore --local
+    run_cmd dvc remote default localstore
+    log_success "Local remote configured at .dvcstore"
+  else
+    log_warning "Remote 'localstore' already exists, skipping"
+  fi
+}
+
+# Update gitignore
+update_gitignore() {
+  log_info "Updating .gitignore..."
+
+  GITIGNORE="$REPO_ROOT/.gitignore"
+
+  # Check if DVC patterns already exist
+  if grep -q "^/\.dvcstore/" "$GITIGNORE" 2>/dev/null; then
+    log_warning "DVC patterns already in .gitignore, skipping"
+    return 0
+  fi
+
+  # Add DVC patterns
+  cat >> "$GITIGNORE" << 'EOF'
+
+# DVC - Data tracked by DVC (pointers in git, data in .dvcstore)
+/.dvcstore/
+/reports/*.csv
+/reports/*.json
+/reports/layer_sweep_*
+/reports/appendices
+/reports/pythia_layer*_vdi_drift*
+/lab/data/corpora
+/lab/data/splits
+/data/lexicons/*.json
+/paper/supplement/*.json
+/paper/supplement/*.csv
+/paper/supplement/cuda_validation
+EOF
+
+  log_success ".gitignore updated with DVC patterns"
+}
+
+# Add DVC tracking
+add_dvc_tracking() {
+  log_info "Adding DVC tracking to data directories..."
+
+  cd "$REPO_ROOT"
+
+  # Track directories and files
+  declare -a DVC_TARGETS=(
+    "lab/data/corpora"
+    "lab/data/splits"
+    "data/lexicons/hedge_booster.json"
+    "reports"
+    "paper/supplement"
+  )
+
+  for target in "${DVC_TARGETS[@]}"; do
+    if [ -e "$target" ]; then
+      log_info "Tracking $target..."
+      run_cmd dvc add "$target"
+
+      # Git add the .dvc file
+      DVC_FILE="${target}.dvc"
+      if [ -f "$DVC_FILE" ]; then
+        run_cmd git add "$DVC_FILE"
+      fi
+    else
+      log_warning "Target not found: $target (skipping)"
+    fi
+  done
+
+  log_success "DVC tracking added"
+}
+
+# Verify tracking
+verify_tracking() {
+  log_info "Verifying DVC tracking..."
+
+  cd "$REPO_ROOT"
+
+  # Check status
+  if [ "$DRY_RUN" = false ]; then
+    dvc status
+  fi
+
+  # List .dvc files
+  log_info "DVC pointer files created:"
+  find . -name "*.dvc" -type f | sed 's|^\./||'
+
+  # Check .dvcstore size
+  if [ -d ".dvcstore" ]; then
+    DVCSTORE_SIZE=$(du -sh .dvcstore | cut -f1)
+    log_info ".dvcstore size: $DVCSTORE_SIZE"
+  fi
+
+  log_success "Verification complete"
+}
+
+# Stage git changes
+stage_git_changes() {
+  log_info "Staging git changes..."
+
+  cd "$REPO_ROOT"
+
+  # Add DVC config files
+  run_cmd git add .dvc/.gitignore .dvc/config .dvc/config.local 2>/dev/null || true
+
+  # Add .gitignore changes
+  run_cmd git add .gitignore
+
+  # Add all .dvc pointer files
+  run_cmd git add "*.dvc" 2>/dev/null || true
+  run_cmd git add "**/*.dvc" 2>/dev/null || true
+
+  log_success "Git changes staged"
+}
+
+# Test data retrieval
+test_retrieval() {
+  log_info "Testing data retrieval (this is a dry-run test)..."
+
+  if [ "$DRY_RUN" = true ]; then
+    log_info "Skipping retrieval test in dry-run mode"
+    return 0
+  fi
+
+  cd "$REPO_ROOT"
+
+  # Create temporary directory
+  TEST_DIR=$(mktemp -d)
+  log_info "Test directory: $TEST_DIR"
+
+  # Try to pull one file
+  log_info "Testing dvc status..."
+  if dvc status; then
+    log_success "DVC status check passed"
+  else
+    log_warning "DVC status check failed (this may be normal if data is already in cache)"
+  fi
+
+  rm -rf "$TEST_DIR"
+}
+
+# Summary
+print_summary() {
+  echo
+  log_info "=========================================="
+  log_info "DVC Migration Summary"
+  log_info "=========================================="
+  echo
+
+  if [ "$DRY_RUN" = true ]; then
+    log_warning "DRY RUN MODE - No changes were made"
+    echo
+    log_info "To execute the migration, run:"
+    log_info "  $0"
+    echo
+  else
+    log_success "Migration completed successfully!"
+    echo
+    log_info "Next steps:"
+    echo "  1. Review changes: git status"
+    echo "  2. Test DVC: dvc status"
+    echo "  3. Commit changes:"
+    echo "     git commit -m 'Add DVC tracking for datasets and results'"
+    echo "  4. Test data retrieval:"
+    echo "     dvc pull"
+    echo "  5. Run tests:"
+    echo "     python smoke_test.py"
+    echo
+    log_info "For more information, see DVC_SETUP.md"
+  fi
+
+  echo
+}
+
+# Main execution
+main() {
+  log_info "Starting DVC migration for tinyLab"
+  echo
+
+  if [ "$DRY_RUN" = true ]; then
+    log_warning "Running in DRY-RUN mode - no changes will be made"
+    echo
+  fi
+
+  check_prerequisites
+  create_backup
+  init_dvc
+  configure_remote
+  update_gitignore
+  add_dvc_tracking
+  verify_tracking
+  stage_git_changes
+  test_retrieval
+  print_summary
+}
+
+# Run main
+main

From 569bbf1f8c37f49670840adb8af21753d7937b79 Mon Sep 17 00:00:00 2001
From: Claude <noreply@anthropic.com>
Date: Tue, 18 Nov 2025 17:13:44 +0000
Subject: [PATCH 2/3] Add Aim experiment tracking integration plan
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Comprehensive plan for adding web-based experiment tracking UI to tinyLab
using Aim (https://aimstack.io/), designed specifically for mechanistic
interpretability research.

## Features

**What Gets Tracked:**
- Core metrics: logit_diff, accuracy, p_drop, KL, calibration
- MI metrics: OV fidelity, QK patterns, activation entropy, geometric signatures
- Head-level analysis: per-head ablation effects, rankings
- Layer-wise metrics: PCA rank, entropy evolution, bottleneck visualization
- Emergence curves: training checkpoint evolution (Pythia)
- Visualizations: attention heatmaps, OV projections, calibration curves

**Benefits:**
- Interactive web UI for browsing all experiments
- Compare runs side-by-side with flexible filtering
- Custom dashboards for specific analyses
- Full Python API for programmatic queries
- Self-hosted, works offline, no vendor lock-in
- Integrates seamlessly with existing DVC workflow

## Components

**Documentation:**
- AIM_INTEGRATION_PLAN.md: Complete architecture and implementation guide
  with 4 implementation phases, code examples, usage guide, and FAQ

**Automation:**
- scripts/setup_aim.sh: Quick setup script for Aim initialization

**Tracking Code (ready to implement):**
- lab/tracking/tracker.py: TinyLabTracker class with methods for:
  - log_metric(), log_head_metrics(), log_layer_metrics()
  - log_ov_fidelity(), log_activation_entropy()
  - log_geometric_metrics(), log_attention_pattern()
  - log_distribution(), log_image(), log_artifact()

**Data Import:**
- scripts/import_to_aim.py: Import existing reports/ data to Aim

**Visualizations:**
- lab/tracking/visualizations.py: MI-specific plots:
  - Layer metric evolution
  - Head effect heatmaps
  - Emergence curves (Pythia checkpoints)
  - OV token projections (2D embeddings)

## Implementation Phases

**Phase 1: Core Integration (30 min)**
- Install Aim, create tracking module, integrate with harness
- Test with single experiment

**Phase 2: Historical Import (1 hour)**
- Import all existing results from reports/ (~300 files)
- Metadata extraction from filenames

**Phase 3: Custom Visualizations (2 hours)**
- Add MI-specific plots and dashboards
- Attention patterns, OV circuits, emergence curves

**Phase 4: DVC Integration (30 min)**
- Ensure .aim/ is gitignored
- Optional: export Aim data for DVC tracking

**Total: ~5-6 hours for complete integration**

## Quick Start

╔════════════════════════════════════════╗
║  tinyLab Aim Tracking Setup           ║
╚════════════════════════════════════════╝

[1/4] Installing Aim...
Collecting aim
  Downloading aim-3.29.1-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (35 kB)
Collecting aim-ui==3.29.1 (from aim)
  Downloading aim-ui-3.29.1.tar.gz (31.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 31.0/31.0 MB 4.1 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting aimrecords==0.0.7 (from aim)
  Downloading aimrecords-0.0.7-py2.py3-none-any.whl.metadata (2.1 kB)
Collecting aimrocks==0.5.* (from aim)
  Downloading aimrocks-0.5.2-cp311-cp311-manylinux_2_24_x86_64.whl.metadata (580 bytes)
Collecting cachetools>=4.0.0 (from aim)
  Downloading cachetools-6.2.2-py3-none-any.whl.metadata (5.6 kB)
Collecting click>=7.0 (from aim)
  Downloading click-8.3.1-py3-none-any.whl.metadata (2.6 kB)
Requirement already satisfied: cryptography>=3.0 in /usr/lib/python3/dist-packages (from aim) (41.0.7)
Collecting filelock<4,>=3.3.0 (from aim)
  Downloading filelock-3.20.0-py3-none-any.whl.metadata (2.1 kB)
Collecting numpy<3,>=1.12.0 (from aim)
  Downloading numpy-2.3.5-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 150.2 MB/s eta 0:00:00
Collecting psutil>=5.6.7 (from aim)
  Downloading psutil-7.1.3-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl.metadata (23 kB)
Collecting RestrictedPython>=5.1 (from aim)
  Downloading restrictedpython-8.1-py3-none-any.whl.metadata (14 kB)
Collecting tqdm>=4.20.0 (from aim)
  Downloading tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.7/57.7 kB 165.1 MB/s eta 0:00:00
Collecting aiofiles>=0.5.0 (from aim)
  Downloading aiofiles-25.1.0-py3-none-any.whl.metadata (6.3 kB)
Collecting alembic<2,>=1.5.0 (from aim)
  Downloading alembic-1.17.2-py3-none-any.whl.metadata (7.2 kB)
Collecting fastapi<1,>=0.69.0 (from aim)
  Downloading fastapi-0.121.2-py3-none-any.whl.metadata (28 kB)
Requirement already satisfied: jinja2<4,>=2.10.0 in /root/.local/lib/python3.11/site-packages (from aim) (3.1.6)
Collecting pytz>=2019.1 (from aim)
  Downloading pytz-2025.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting SQLAlchemy>=1.4.1 (from aim)
  Downloading sqlalchemy-2.0.44-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.5 kB)
Collecting uvicorn<1,>=0.12.0 (from aim)
  Downloading uvicorn-0.38.0-py3-none-any.whl.metadata (6.8 kB)
Collecting Pillow>=8.0.0 (from aim)
  Downloading pillow-12.0.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (8.8 kB)
Requirement already satisfied: packaging>=15.0 in /usr/lib/python3/dist-packages (from aim) (24.0)
Requirement already satisfied: python-dateutil in /root/.local/lib/python3.11/site-packages (from aim) (2.9.0.post0)
Requirement already satisfied: requests in /root/.local/lib/python3.11/site-packages (from aim) (2.32.5)
Collecting watchdog (from aim)
  Downloading watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl.metadata (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.3/44.3 kB 126.3 MB/s eta 0:00:00
Collecting websockets (from aim)
  Downloading websockets-15.0.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB)
Collecting boto3 (from aim)
  Downloading boto3-1.40.75-py3-none-any.whl.metadata (6.8 kB)
Collecting base58==2.0.1 (from aimrecords==0.0.7->aim)
  Downloading base58-2.0.1-py3-none-any.whl.metadata (540 bytes)
Collecting Mako (from alembic<2,>=1.5.0->aim)
  Downloading mako-1.3.10-py3-none-any.whl.metadata (2.9 kB)
Collecting typing-extensions>=4.12 (from alembic<2,>=1.5.0->aim)
  Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Collecting starlette<0.50.0,>=0.40.0 (from fastapi<1,>=0.69.0->aim)
  Downloading starlette-0.49.3-py3-none-any.whl.metadata (6.4 kB)
Collecting pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 (from fastapi<1,>=0.69.0->aim)
  Downloading pydantic-2.12.4-py3-none-any.whl.metadata (89 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.9/89.9 kB 7.2 MB/s eta 0:00:00
Collecting annotated-doc>=0.0.2 (from fastapi<1,>=0.69.0->aim)
  Downloading annotated_doc-0.0.4-py3-none-any.whl.metadata (6.6 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /root/.local/lib/python3.11/site-packages (from jinja2<4,>=2.10.0->aim) (3.0.3)
Collecting greenlet>=1 (from SQLAlchemy>=1.4.1->aim)
  Downloading greenlet-3.2.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (4.1 kB)
Collecting h11>=0.8 (from uvicorn<1,>=0.12.0->aim)
  Downloading h11-0.16.0-py3-none-any.whl.metadata (8.3 kB)
Collecting botocore<1.41.0,>=1.40.75 (from boto3->aim)
  Downloading botocore-1.40.75-py3-none-any.whl.metadata (5.9 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3->aim)
  Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.15.0,>=0.14.0 (from boto3->aim)
  Downloading s3transfer-0.14.0-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->aim) (1.16.0)
Requirement already satisfied: charset_normalizer<4,>=2 in /root/.local/lib/python3.11/site-packages (from requests->aim) (3.4.4)
Requirement already satisfied: idna<4,>=2.5 in /root/.local/lib/python3.11/site-packages (from requests->aim) (3.11)
Requirement already satisfied: urllib3<3,>=1.21.1 in /root/.local/lib/python3.11/site-packages (from requests->aim) (2.5.0)
Requirement already satisfied: certifi>=2017.4.17 in /root/.local/lib/python3.11/site-packages (from requests->aim) (2025.11.12)
Collecting annotated-types>=0.6.0 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi<1,>=0.69.0->aim)
  Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)
Collecting pydantic-core==2.41.5 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi<1,>=0.69.0->aim)
  Downloading pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.3 kB)
Collecting typing-inspection>=0.4.2 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi<1,>=0.69.0->aim)
  Downloading typing_inspection-0.4.2-py3-none-any.whl.metadata (2.6 kB)
Collecting anyio<5,>=3.6.2 (from starlette<0.50.0,>=0.40.0->fastapi<1,>=0.69.0->aim)
  Downloading anyio-4.11.0-py3-none-any.whl.metadata (4.1 kB)
Collecting sniffio>=1.1 (from anyio<5,>=3.6.2->starlette<0.50.0,>=0.40.0->fastapi<1,>=0.69.0->aim)
  Downloading sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Downloading aim-3.29.1-cp311-cp311-manylinux_2_28_x86_64.whl (7.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.5/7.5 MB 3.5 MB/s eta 0:00:00
Downloading aimrecords-0.0.7-py2.py3-none-any.whl (28 kB)
Downloading aimrocks-0.5.2-cp311-cp311-manylinux_2_24_x86_64.whl (6.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/6.2 MB 5.5 MB/s eta 0:00:00
Downloading base58-2.0.1-py3-none-any.whl (4.3 kB)
Downloading aiofiles-25.1.0-py3-none-any.whl (14 kB)
Downloading alembic-1.17.2-py3-none-any.whl (248 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 248.6/248.6 kB 6.8 MB/s eta 0:00:00
Downloading cachetools-6.2.2-py3-none-any.whl (11 kB)
Downloading click-8.3.1-py3-none-any.whl (108 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.3/108.3 kB 8.5 MB/s eta 0:00:00
Downloading fastapi-0.121.2-py3-none-any.whl (109 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.2/109.2 kB 8.5 MB/s eta 0:00:00
Downloading filelock-3.20.0-py3-none-any.whl (16 kB)
Downloading numpy-2.3.5-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/16.9 MB 3.6 MB/s eta 0:00:00
Downloading pillow-12.0.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (7.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.0/7.0 MB 3.7 MB/s eta 0:00:00
Downloading psutil-7.1.3-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl (263 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 263.3/263.3 kB 4.6 MB/s eta 0:00:00
Downloading pytz-2025.2-py2.py3-none-any.whl (509 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 509.2/509.2 kB 6.0 MB/s eta 0:00:00
Downloading restrictedpython-8.1-py3-none-any.whl (27 kB)
Downloading sqlalchemy-2.0.44-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 5.6 MB/s eta 0:00:00
Downloading tqdm-4.67.1-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 kB 6.7 MB/s eta 0:00:00
Downloading uvicorn-0.38.0-py3-none-any.whl (68 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.1/68.1 kB 5.5 MB/s eta 0:00:00
Downloading boto3-1.40.75-py3-none-any.whl (139 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.4/139.4 kB 5.7 MB/s eta 0:00:00
Downloading watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl (79 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.1/79.1 kB 5.8 MB/s eta 0:00:00
Downloading websockets-15.0.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (182 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 182.3/182.3 kB 7.3 MB/s eta 0:00:00
Downloading annotated_doc-0.0.4-py3-none-any.whl (5.3 kB)
Downloading botocore-1.40.75-py3-none-any.whl (14.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 4.3 MB/s eta 0:00:00
Downloading greenlet-3.2.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (587 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 587.7/587.7 kB 6.0 MB/s eta 0:00:00
Downloading h11-0.16.0-py3-none-any.whl (37 kB)
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading pydantic-2.12.4-py3-none-any.whl (463 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 463.4/463.4 kB 5.7 MB/s eta 0:00:00
Downloading pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 3.7 MB/s eta 0:00:00
Downloading s3transfer-0.14.0-py3-none-any.whl (85 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.7/85.7 kB 7.2 MB/s eta 0:00:00
Downloading starlette-0.49.3-py3-none-any.whl (74 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.3/74.3 kB 6.6 MB/s eta 0:00:00
Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 kB 100.5 MB/s eta 0:00:00
Downloading mako-1.3.10-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 kB 6.6 MB/s eta 0:00:00
Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB)
Downloading anyio-4.11.0-py3-none-any.whl (109 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.1/109.1 kB 8.4 MB/s eta 0:00:00
Downloading typing_inspection-0.4.2-py3-none-any.whl (14 kB)
Downloading sniffio-1.3.1-py3-none-any.whl (10 kB)
Building wheels for collected packages: aim-ui
  Building wheel for aim-ui (setup.py): started
  Building wheel for aim-ui (setup.py): finished with status 'error'
  Running setup.py clean for aim-ui
Failed to build aim-ui
Collecting aim
  Downloading aim-3.29.1-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (35 kB)
Collecting aim-ui==3.29.1 (from aim)
  Downloading aim-ui-3.29.1.tar.gz (31.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 31.0/31.0 MB 3.9 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting aimrecords==0.0.7 (from aim)
  Downloading aimrecords-0.0.7-py2.py3-none-any.whl.metadata (2.1 kB)
Collecting aimrocks==0.5.* (from aim)
  Downloading aimrocks-0.5.2-cp311-cp311-manylinux_2_24_x86_64.whl.metadata (580 bytes)
Collecting cachetools>=4.0.0 (from aim)
  Downloading cachetools-6.2.2-py3-none-any.whl.metadata (5.6 kB)
Collecting click>=7.0 (from aim)
  Downloading click-8.3.1-py3-none-any.whl.metadata (2.6 kB)
Requirement already satisfied: cryptography>=3.0 in /usr/lib/python3/dist-packages (from aim) (41.0.7)
Collecting filelock<4,>=3.3.0 (from aim)
  Downloading filelock-3.20.0-py3-none-any.whl.metadata (2.1 kB)
Collecting numpy<3,>=1.12.0 (from aim)
  Downloading numpy-2.3.5-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 150.5 MB/s eta 0:00:00
Collecting psutil>=5.6.7 (from aim)
  Downloading psutil-7.1.3-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl.metadata (23 kB)
Collecting RestrictedPython>=5.1 (from aim)
  Downloading restrictedpython-8.1-py3-none-any.whl.metadata (14 kB)
Collecting tqdm>=4.20.0 (from aim)
  Downloading tqdm-4.67.1-py3-none-any.whl.metadata (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.7/57.7 kB 177.9 MB/s eta 0:00:00
Collecting aiofiles>=0.5.0 (from aim)
  Downloading aiofiles-25.1.0-py3-none-any.whl.metadata (6.3 kB)
Collecting alembic<2,>=1.5.0 (from aim)
  Downloading alembic-1.17.2-py3-none-any.whl.metadata (7.2 kB)
Collecting fastapi<1,>=0.69.0 (from aim)
  Downloading fastapi-0.121.2-py3-none-any.whl.metadata (28 kB)
Requirement already satisfied: jinja2<4,>=2.10.0 in /root/.local/lib/python3.11/site-packages (from aim) (3.1.6)
Collecting pytz>=2019.1 (from aim)
  Downloading pytz-2025.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting SQLAlchemy>=1.4.1 (from aim)
  Downloading sqlalchemy-2.0.44-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (9.5 kB)
Collecting uvicorn<1,>=0.12.0 (from aim)
  Downloading uvicorn-0.38.0-py3-none-any.whl.metadata (6.8 kB)
Collecting Pillow>=8.0.0 (from aim)
  Downloading pillow-12.0.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl.metadata (8.8 kB)
Requirement already satisfied: packaging>=15.0 in /usr/lib/python3/dist-packages (from aim) (24.0)
Requirement already satisfied: python-dateutil in /root/.local/lib/python3.11/site-packages (from aim) (2.9.0.post0)
Requirement already satisfied: requests in /root/.local/lib/python3.11/site-packages (from aim) (2.32.5)
Collecting watchdog (from aim)
  Downloading watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl.metadata (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.3/44.3 kB 148.5 MB/s eta 0:00:00
Collecting websockets (from aim)
  Downloading websockets-15.0.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.8 kB)
Collecting boto3 (from aim)
  Downloading boto3-1.40.75-py3-none-any.whl.metadata (6.8 kB)
Collecting base58==2.0.1 (from aimrecords==0.0.7->aim)
  Downloading base58-2.0.1-py3-none-any.whl.metadata (540 bytes)
Collecting Mako (from alembic<2,>=1.5.0->aim)
  Downloading mako-1.3.10-py3-none-any.whl.metadata (2.9 kB)
Collecting typing-extensions>=4.12 (from alembic<2,>=1.5.0->aim)
  Downloading typing_extensions-4.15.0-py3-none-any.whl.metadata (3.3 kB)
Collecting starlette<0.50.0,>=0.40.0 (from fastapi<1,>=0.69.0->aim)
  Downloading starlette-0.49.3-py3-none-any.whl.metadata (6.4 kB)
Collecting pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 (from fastapi<1,>=0.69.0->aim)
  Downloading pydantic-2.12.4-py3-none-any.whl.metadata (89 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.9/89.9 kB 8.9 MB/s eta 0:00:00
Collecting annotated-doc>=0.0.2 (from fastapi<1,>=0.69.0->aim)
  Downloading annotated_doc-0.0.4-py3-none-any.whl.metadata (6.6 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /root/.local/lib/python3.11/site-packages (from jinja2<4,>=2.10.0->aim) (3.0.3)
Collecting greenlet>=1 (from SQLAlchemy>=1.4.1->aim)
  Downloading greenlet-3.2.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl.metadata (4.1 kB)
Collecting h11>=0.8 (from uvicorn<1,>=0.12.0->aim)
  Downloading h11-0.16.0-py3-none-any.whl.metadata (8.3 kB)
Collecting botocore<1.41.0,>=1.40.75 (from boto3->aim)
  Downloading botocore-1.40.75-py3-none-any.whl.metadata (5.9 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3->aim)
  Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.15.0,>=0.14.0 (from boto3->aim)
  Downloading s3transfer-0.14.0-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil->aim) (1.16.0)
Requirement already satisfied: charset_normalizer<4,>=2 in /root/.local/lib/python3.11/site-packages (from requests->aim) (3.4.4)
Requirement already satisfied: idna<4,>=2.5 in /root/.local/lib/python3.11/site-packages (from requests->aim) (3.11)
Requirement already satisfied: urllib3<3,>=1.21.1 in /root/.local/lib/python3.11/site-packages (from requests->aim) (2.5.0)
Requirement already satisfied: certifi>=2017.4.17 in /root/.local/lib/python3.11/site-packages (from requests->aim) (2025.11.12)
Collecting annotated-types>=0.6.0 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi<1,>=0.69.0->aim)
  Downloading annotated_types-0.7.0-py3-none-any.whl.metadata (15 kB)
Collecting pydantic-core==2.41.5 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi<1,>=0.69.0->aim)
  Downloading pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.3 kB)
Collecting typing-inspection>=0.4.2 (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4->fastapi<1,>=0.69.0->aim)
  Downloading typing_inspection-0.4.2-py3-none-any.whl.metadata (2.6 kB)
Collecting anyio<5,>=3.6.2 (from starlette<0.50.0,>=0.40.0->fastapi<1,>=0.69.0->aim)
  Downloading anyio-4.11.0-py3-none-any.whl.metadata (4.1 kB)
Collecting sniffio>=1.1 (from anyio<5,>=3.6.2->starlette<0.50.0,>=0.40.0->fastapi<1,>=0.69.0->aim)
  Downloading sniffio-1.3.1-py3-none-any.whl.metadata (3.9 kB)
Downloading aim-3.29.1-cp311-cp311-manylinux_2_28_x86_64.whl (7.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.5/7.5 MB 6.3 MB/s eta 0:00:00
Downloading aimrecords-0.0.7-py2.py3-none-any.whl (28 kB)
Downloading aimrocks-0.5.2-cp311-cp311-manylinux_2_24_x86_64.whl (6.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.2/6.2 MB 4.0 MB/s eta 0:00:00
Downloading base58-2.0.1-py3-none-any.whl (4.3 kB)
Downloading aiofiles-25.1.0-py3-none-any.whl (14 kB)
Downloading alembic-1.17.2-py3-none-any.whl (248 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 248.6/248.6 kB 4.8 MB/s eta 0:00:00
Downloading cachetools-6.2.2-py3-none-any.whl (11 kB)
Downloading click-8.3.1-py3-none-any.whl (108 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 108.3/108.3 kB 9.3 MB/s eta 0:00:00
Downloading fastapi-0.121.2-py3-none-any.whl (109 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.2/109.2 kB 9.2 MB/s eta 0:00:00
Downloading filelock-3.20.0-py3-none-any.whl (16 kB)
Downloading numpy-2.3.5-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (16.9 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/16.9 MB 3.9 MB/s eta 0:00:00
Downloading pillow-12.0.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl (7.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.0/7.0 MB 6.0 MB/s eta 0:00:00
Downloading psutil-7.1.3-cp36-abi3-manylinux2010_x86_64.manylinux_2_12_x86_64.manylinux_2_28_x86_64.whl (263 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 263.3/263.3 kB 5.9 MB/s eta 0:00:00
Downloading pytz-2025.2-py2.py3-none-any.whl (509 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 509.2/509.2 kB 4.2 MB/s eta 0:00:00
Downloading restrictedpython-8.1-py3-none-any.whl (27 kB)
Downloading sqlalchemy-2.0.44-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 3.9 MB/s eta 0:00:00
Downloading tqdm-4.67.1-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 kB 7.1 MB/s eta 0:00:00
Downloading uvicorn-0.38.0-py3-none-any.whl (68 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.1/68.1 kB 6.0 MB/s eta 0:00:00
Downloading boto3-1.40.75-py3-none-any.whl (139 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.4/139.4 kB 6.3 MB/s eta 0:00:00
Downloading watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl (79 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.1/79.1 kB 6.9 MB/s eta 0:00:00
Downloading websockets-15.0.1-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (182 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 182.3/182.3 kB 5.6 MB/s eta 0:00:00
Downloading annotated_doc-0.0.4-py3-none-any.whl (5.3 kB)
Downloading botocore-1.40.75-py3-none-any.whl (14.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.1/14.1 MB 6.0 MB/s eta 0:00:00
Downloading greenlet-3.2.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (587 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 587.7/587.7 kB 6.4 MB/s eta 0:00:00
Downloading h11-0.16.0-py3-none-any.whl (37 kB)
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading pydantic-2.12.4-py3-none-any.whl (463 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 463.4/463.4 kB 6.0 MB/s eta 0:00:00
Downloading pydantic_core-2.41.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 6.1 MB/s eta 0:00:00
Downloading s3transfer-0.14.0-py3-none-any.whl (85 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.7/85.7 kB 7.7 MB/s eta 0:00:00
Downloading starlette-0.49.3-py3-none-any.whl (74 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 74.3/74.3 kB 6.6 MB/s eta 0:00:00
Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 kB 274.1 MB/s eta 0:00:00
Downloading mako-1.3.10-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.5/78.5 kB 6.8 MB/s eta 0:00:00
Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB)
Downloading anyio-4.11.0-py3-none-any.whl (109 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.1/109.1 kB 9.0 MB/s eta 0:00:00
Downloading typing_inspection-0.4.2-py3-none-any.whl (14 kB)
Downloading sniffio-1.3.1-py3-none-any.whl (10 kB)
Building wheels for collected packages: aim-ui
  Building wheel for aim-ui (setup.py): started
  Building wheel for aim-ui (setup.py): finished with status 'error'
  Running setup.py clean for aim-ui
Failed to build aim-ui

## Use Cases

**Research:**
- Compare suppressor effects across models (GPT-2, Mistral, Pythia)
- Track emergence of circuits during training
- Visualize layer-wise metric evolution
- Identify top-performing heads interactively

**Collaboration:**
- Share experiment URLs with team
- Create saved dashboards for specific analyses
- Export metrics for papers (LaTeX tables)

**Debugging:**
- Quickly spot anomalous runs
- Compare failed vs successful experiments
- Track system metrics (VRAM, timing)

## Integration with DVC

Aim tracking (.aim/) is local and regenerable from DVC-tracked data:
- DVC tracks: raw data, configs, results
- Aim tracks: experiment metadata, metrics, visualizations
- Import script regenerates Aim DB from DVC-pulled reports/

This keeps git clean while enabling powerful experiment exploration.

See AIM_INTEGRATION_PLAN.md for complete implementation guide.
---
 AIM_INTEGRATION_PLAN.md | 1066 +++++++++++++++++++++++++++++++++++++++
 scripts/setup_aim.sh    |   74 +++
 2 files changed, 1140 insertions(+)
 create mode 100644 AIM_INTEGRATION_PLAN.md
 create mode 100755 scripts/setup_aim.sh

diff --git a/AIM_INTEGRATION_PLAN.md b/AIM_INTEGRATION_PLAN.md
new file mode 100644
index 0000000..fef5da7
--- /dev/null
+++ b/AIM_INTEGRATION_PLAN.md
@@ -0,0 +1,1066 @@
+# Aim Experiment Tracking Integration for tinyLab
+
+**Status:** Design Complete - Ready for Implementation
+**Date:** 2025-11-18
+**Purpose:** Add comprehensive experiment tracking with web UI for mechanistic interpretability research
+
+---
+
+## Overview
+
+This document outlines the integration of [Aim](https://aimstack.io/) experiment tracking into tinyLab. Aim will provide an interactive web UI to browse experiments, compare runs, and visualize mechanistic interpretability metrics in real-time.
+
+### Why Aim?
+
+- **Self-hosted** - No cloud dependencies, works offline
+- **Python-native** - Easy integration with existing codebase
+- **Rich visualizations** - Interactive plots, comparisons, filtering
+- **Flexible** - Supports custom metrics, images, text, distributions
+- **Fast** - Efficient storage and querying
+- **Open source** - MIT license, no vendor lock-in
+
+### What Gets Tracked
+
+```
+Run Metadata         Core Metrics              MI Metrics                    Artifacts
+├── model_name       ├── logit_diff           ├── ov_fidelity_by_layer     ├── attention_heatmaps
+├── condition        ├── accuracy             ├── qk_pattern_strength      ├── ov_projections
+├── probe_type       ├── p_drop               ├── activation_entropy       ├── calibration_curves
+├── layers           ├── kl_divergence        ├── geometric_curvature      ├── confusion_matrices
+├── heads            ├── calibration_ece      ├── pca_rank_by_layer        ├── token_clouds
+├── seed             ├── mediation_fraction   ├── path_patching_effects    └── trajectory_plots
+├── git_commit       └── bootstrap_ci         └── emergence_curves
+├── timestamp
+└── device
+```
+
+---
+
+## Architecture
+
+### Directory Structure
+
+```
+tinyLab/
+├── .aim/                       # Aim storage (gitignored)
+│   ├── meta/                   # Metadata index
+│   ├── runs/                   # Run data (metrics, logs)
+│   └── seqs/                   # Sequence storage
+│
+├── lab/
+│   ├── tracking/               # NEW: Aim integration code
+│   │   ├── __init__.py
+│   │   ├── tracker.py          # Main tracking class
+│   │   ├── metrics.py          # Metric definitions
+│   │   ├── visualizations.py  # Custom plots
+│   │   └── migrate.py          # Import existing results
+│   │
+│   ├── harness.py              # MODIFIED: Add tracking hooks
+│   └── configs/                # EXISTING: Experiment configs
+│
+├── scripts/
+│   ├── import_to_aim.py        # Import historical results
+│   └── launch_aim_ui.sh        # Start Aim web UI
+│
+└── docs/
+    └── AIM_USAGE.md            # User guide for Aim UI
+```
+
+### Data Flow
+
+```
+Experiment Run (harness.py)
+    ↓
+TinyLabTracker.log_metrics()
+    ↓
+Aim Run Storage (.aim/)
+    ↓
+Aim UI (http://localhost:43800)
+    ↓
+Interactive Visualizations
+```
+
+---
+
+## Implementation Plan
+
+### Phase 1: Core Integration (30 min)
+
+**Goal:** Basic tracking of runs with metadata and core metrics
+
+#### 1. Install Aim
+
+```bash
+pip install aim
+```
+
+#### 2. Create Tracking Module
+
+**`lab/tracking/__init__.py`:**
+```python
+"""Experiment tracking with Aim."""
+from .tracker import TinyLabTracker
+
+__all__ = ['TinyLabTracker']
+```
+
+**`lab/tracking/tracker.py`:**
+```python
+"""Main tracking class for tinyLab experiments."""
+from aim import Run
+from pathlib import Path
+from typing import Dict, Any, Optional
+import json
+
+class TinyLabTracker:
+    """
+    Wrapper around Aim for mechanistic interpretability experiments.
+
+    Example usage:
+        tracker = TinyLabTracker(
+            experiment_name="h1_suppressor_sweep",
+            config=config_dict,
+            tags=["gpt2-medium", "facts", "layer0"]
+        )
+
+        # Log metrics
+        tracker.log_metric("logit_diff", 2.45, step=0)
+        tracker.log_metric("accuracy", 0.89, step=0)
+
+        # Log custom MI metrics
+        tracker.log_ov_fidelity(ov_scores_by_layer, step=0)
+
+        # Log artifacts
+        tracker.log_attention_pattern(attn_matrix, head=(0, 2))
+
+        # Finish
+        tracker.finish()
+    """
+
+    def __init__(
+        self,
+        experiment_name: str,
+        config: Dict[str, Any],
+        tags: Optional[list] = None,
+        repo_path: Optional[str] = None
+    ):
+        """
+        Initialize tracker for an experiment run.
+
+        Args:
+            experiment_name: Name of experiment (e.g., "h1_cross_condition")
+            config: Full experiment configuration dict
+            tags: List of tags for filtering (e.g., ["gpt2-medium", "facts"])
+            repo_path: Path to .aim directory (default: project root)
+        """
+        self.experiment_name = experiment_name
+        self.config = config
+
+        # Initialize Aim run
+        self.run = Run(
+            repo=repo_path,
+            experiment=experiment_name,
+            tags=tags or []
+        )
+
+        # Log all config as hyperparameters
+        self.run['hparams'] = config
+
+        # Log key metadata
+        self.run['model_name'] = config.get('model_name', 'unknown')
+        self.run['condition'] = config.get('tag', 'unknown')
+        self.run['probe_type'] = config.get('probe', 'unknown')
+        self.run['device'] = config.get('device', 'unknown')
+        self.run['seed'] = config.get('seed', None)
+
+        # Git info
+        import git
+        try:
+            repo = git.Repo(search_parent_directories=True)
+            self.run['git_commit'] = repo.head.commit.hexsha[:8]
+            self.run['git_branch'] = repo.active_branch.name
+        except:
+            pass
+
+    def log_metric(self, name: str, value: float, step: int = 0, context: Optional[Dict] = None):
+        """
+        Log a scalar metric.
+
+        Args:
+            name: Metric name (e.g., "logit_diff")
+            value: Metric value
+            step: Step/iteration (0 for final metrics)
+            context: Additional context (e.g., {"head": "0:2"})
+        """
+        self.run.track(value, name=name, step=step, context=context or {})
+
+    def log_metrics_dict(self, metrics: Dict[str, float], step: int = 0, prefix: str = ""):
+        """
+        Log multiple metrics at once.
+
+        Args:
+            metrics: Dict of {metric_name: value}
+            step: Step/iteration
+            prefix: Prefix to add to all metric names
+        """
+        for name, value in metrics.items():
+            full_name = f"{prefix}/{name}" if prefix else name
+            self.log_metric(full_name, value, step=step)
+
+    def log_head_metrics(self, head: tuple, metrics: Dict[str, float], step: int = 0):
+        """
+        Log metrics for a specific attention head.
+
+        Args:
+            head: Tuple of (layer, head_idx)
+            metrics: Dict of {metric_name: value}
+            step: Step/iteration
+        """
+        context = {"layer": head[0], "head": head[1]}
+        for name, value in metrics.items():
+            self.run.track(value, name=name, step=step, context=context)
+
+    def log_layer_metrics(self, layer: int, metrics: Dict[str, float], step: int = 0):
+        """
+        Log metrics for a specific layer.
+
+        Args:
+            layer: Layer index
+            metrics: Dict of {metric_name: value}
+            step: Step/iteration
+        """
+        context = {"layer": layer}
+        for name, value in metrics.items():
+            self.run.track(value, name=name, step=step, context=context)
+
+    def log_ov_fidelity(self, fidelity_by_layer: Dict[int, float], step: int = 0):
+        """
+        Log OV circuit fidelity across layers.
+
+        Args:
+            fidelity_by_layer: {layer_idx: fidelity_score}
+            step: Step/iteration
+        """
+        for layer, fidelity in fidelity_by_layer.items():
+            self.run.track(
+                fidelity,
+                name="ov_fidelity",
+                step=step,
+                context={"layer": layer}
+            )
+
+    def log_activation_entropy(
+        self,
+        layer: int,
+        entropy: float,
+        entropy_type: str = "subspace",
+        step: int = 0
+    ):
+        """
+        Log activation entropy for a layer.
+
+        Args:
+            layer: Layer index
+            entropy: Entropy value
+            entropy_type: Type of entropy ("subspace", "diagonal", "per_token")
+            step: Step/iteration
+        """
+        self.run.track(
+            entropy,
+            name=f"activation_entropy_{entropy_type}",
+            step=step,
+            context={"layer": layer}
+        )
+
+    def log_geometric_metrics(
+        self,
+        curvature: float,
+        output_entropy: float,
+        step: int = 0,
+        phase: str = "final"
+    ):
+        """
+        Log geometric signature metrics.
+
+        Args:
+            curvature: Trajectory curvature
+            output_entropy: Output distribution entropy
+            step: Step/iteration
+            phase: Phase of trajectory ("early", "mid", "final")
+        """
+        self.run.track(curvature, name="curvature", step=step, context={"phase": phase})
+        self.run.track(output_entropy, name="output_entropy", step=step, context={"phase": phase})
+
+    def log_image(self, name: str, image, step: int = 0, context: Optional[Dict] = None):
+        """
+        Log an image (attention pattern, plot, etc.).
+
+        Args:
+            name: Image name
+            image: PIL Image, numpy array, or matplotlib figure
+            step: Step/iteration
+            context: Additional context
+        """
+        from aim import Image
+        self.run.track(Image(image), name=name, step=step, context=context or {})
+
+    def log_attention_pattern(self, pattern, layer: int, head: int, step: int = 0):
+        """
+        Log attention pattern heatmap.
+
+        Args:
+            pattern: Attention matrix (numpy array or matplotlib figure)
+            layer: Layer index
+            head: Head index
+            step: Step/iteration
+        """
+        import matplotlib.pyplot as plt
+        from aim import Image
+
+        # If pattern is numpy array, create heatmap
+        if hasattr(pattern, 'shape'):
+            fig, ax = plt.subplots(figsize=(8, 6))
+            im = ax.imshow(pattern, cmap='viridis', aspect='auto')
+            ax.set_title(f'Attention Pattern L{layer}H{head}')
+            ax.set_xlabel('Key Position')
+            ax.set_ylabel('Query Position')
+            plt.colorbar(im, ax=ax)
+            self.run.track(
+                Image(fig),
+                name="attention_pattern",
+                step=step,
+                context={"layer": layer, "head": head}
+            )
+            plt.close(fig)
+        else:
+            # Assume it's already a figure
+            self.run.track(
+                Image(pattern),
+                name="attention_pattern",
+                step=step,
+                context={"layer": layer, "head": head}
+            )
+
+    def log_distribution(self, name: str, values, step: int = 0, context: Optional[Dict] = None):
+        """
+        Log a distribution of values.
+
+        Args:
+            name: Distribution name
+            values: Array of values
+            step: Step/iteration
+            context: Additional context
+        """
+        from aim import Distribution
+        self.run.track(
+            Distribution(values),
+            name=name,
+            step=step,
+            context=context or {}
+        )
+
+    def log_text(self, name: str, text: str, step: int = 0):
+        """
+        Log text (e.g., model output, errors).
+
+        Args:
+            name: Text identifier
+            text: Text content
+            step: Step/iteration
+        """
+        from aim import Text
+        self.run.track(Text(text), name=name, step=step)
+
+    def log_artifact(self, name: str, artifact: Any):
+        """
+        Log arbitrary Python object as artifact.
+
+        Args:
+            name: Artifact name
+            artifact: Any JSON-serializable object
+        """
+        self.run[name] = artifact
+
+    def finish(self, final_metrics: Optional[Dict[str, float]] = None):
+        """
+        Finalize the run.
+
+        Args:
+            final_metrics: Optional final metrics to log
+        """
+        if final_metrics:
+            self.log_metrics_dict(final_metrics, prefix="final")
+
+        self.run.close()
+```
+
+#### 3. Integrate with Harness
+
+**Modify `lab/harness.py`:**
+
+```python
+# At top of file
+from lab.tracking import TinyLabTracker
+
+# In the main experiment function:
+def run_experiment(config_path: str):
+    # Load config
+    config = load_config(config_path)
+
+    # Initialize tracker
+    tracker = TinyLabTracker(
+        experiment_name=config.get('experiment', 'unnamed'),
+        config=config,
+        tags=[
+            config['model_name'],
+            config.get('tag', 'unknown'),
+            f"layer{config.get('target_layer', 0)}"
+        ]
+    )
+
+    try:
+        # Run experiment
+        results = run_ablation_sweep(config)
+
+        # Log results
+        for head, metrics in results.items():
+            tracker.log_head_metrics(
+                head=head,
+                metrics={
+                    'logit_diff': metrics['ld'],
+                    'accuracy': metrics['acc'],
+                    'p_drop': metrics['p_drop'],
+                    'kl_divergence': metrics['kl']
+                }
+            )
+
+        # Log aggregate metrics
+        tracker.log_metrics_dict({
+            'mean_logit_diff': np.mean([m['ld'] for m in results.values()]),
+            'max_logit_diff': np.max([m['ld'] for m in results.values()]),
+            'top_head_ld': sorted(results.items(), key=lambda x: x[1]['ld'])[-1][1]['ld']
+        })
+
+    finally:
+        tracker.finish()
+```
+
+#### 4. Start Aim UI
+
+```bash
+# Launch web UI
+aim up
+
+# Opens at http://localhost:43800
+```
+
+---
+
+### Phase 2: Historical Data Import (1 hour)
+
+**Goal:** Import existing results from `reports/` into Aim
+
+**`scripts/import_to_aim.py`:**
+```python
+#!/usr/bin/env python3
+"""
+Import historical tinyLab results into Aim.
+
+Usage:
+    python scripts/import_to_aim.py
+    python scripts/import_to_aim.py --reports-dir reports/
+"""
+import argparse
+import json
+from pathlib import Path
+from aim import Run
+import re
+
+def parse_filename(filename: str):
+    """Extract metadata from filename."""
+    # Examples:
+    # gpt2m_facts_ranking.csv
+    # mistral_cf_l0_ranking.csv
+    # h1_head_rank_stats.json
+
+    parts = filename.stem.split('_')
+    metadata = {}
+
+    # Extract model
+    if 'gpt2' in filename.stem:
+        if 'gpt2m' in filename.stem:
+            metadata['model'] = 'gpt2-medium'
+        elif 'gpt2l' in filename.stem:
+            metadata['model'] = 'gpt2-large'
+        else:
+            metadata['model'] = 'gpt2'
+    elif 'mistral' in filename.stem:
+        metadata['model'] = 'mistral-7b'
+    elif 'pythia' in filename.stem:
+        metadata['model'] = 'pythia'
+
+    # Extract condition
+    conditions = ['facts', 'cf', 'logic', 'neg', 'counterfactual', 'negation', 'logical']
+    for cond in conditions:
+        if cond in filename.stem:
+            metadata['condition'] = cond
+            break
+
+    # Extract hypothesis
+    h_match = re.search(r'h(\d+)', filename.stem)
+    if h_match:
+        metadata['hypothesis'] = f"H{h_match.group(1)}"
+
+    return metadata
+
+def import_head_rankings(csv_path: Path, repo_path: str = None):
+    """Import head ranking CSV."""
+    import pandas as pd
+
+    metadata = parse_filename(csv_path)
+
+    run = Run(
+        repo=repo_path,
+        experiment=f"imported_{metadata.get('hypothesis', 'ranking')}",
+        tags=['imported', 'historical'] + list(metadata.values())
+    )
+
+    # Log metadata
+    run['source_file'] = str(csv_path)
+    run['imported'] = True
+    for k, v in metadata.items():
+        run[k] = v
+
+    # Load and log data
+    df = pd.read_csv(csv_path)
+
+    for idx, row in df.iterrows():
+        layer = row.get('layer', row.get('Layer', 0))
+        head = row.get('head', row.get('Head', idx))
+
+        context = {"layer": int(layer), "head": int(head)}
+
+        # Log available metrics
+        for col in df.columns:
+            if col.lower() in ['layer', 'head', 'rank']:
+                continue
+            try:
+                value = float(row[col])
+                run.track(value, name=col.lower(), step=0, context=context)
+            except:
+                pass
+
+    run.close()
+    print(f"✓ Imported {csv_path.name}")
+
+def import_json_metrics(json_path: Path, repo_path: str = None):
+    """Import JSON metric file."""
+    metadata = parse_filename(json_path)
+
+    run = Run(
+        repo=repo_path,
+        experiment=f"imported_{metadata.get('hypothesis', 'metrics')}",
+        tags=['imported', 'historical'] + list(metadata.values())
+    )
+
+    # Log metadata
+    run['source_file'] = str(json_path)
+    run['imported'] = True
+    for k, v in metadata.items():
+        run[k] = v
+
+    # Load data
+    with open(json_path) as f:
+        data = json.load(f)
+
+    # Log all metrics
+    def log_nested(obj, prefix=""):
+        """Recursively log nested dict."""
+        if isinstance(obj, dict):
+            for key, val in obj.items():
+                new_prefix = f"{prefix}/{key}" if prefix else key
+                log_nested(val, new_prefix)
+        elif isinstance(obj, (int, float)):
+            run.track(float(obj), name=prefix, step=0)
+        elif isinstance(obj, list) and all(isinstance(x, (int, float)) for x in obj):
+            # Log as distribution
+            from aim import Distribution
+            run.track(Distribution(obj), name=prefix, step=0)
+
+    log_nested(data)
+
+    run.close()
+    print(f"✓ Imported {json_path.name}")
+
+def main():
+    parser = argparse.ArgumentParser(description="Import tinyLab results to Aim")
+    parser.add_argument('--reports-dir', default='reports/', help='Path to reports directory')
+    parser.add_argument('--repo', default=None, help='Path to .aim directory')
+    args = parser.parse_args()
+
+    reports_dir = Path(args.reports_dir)
+
+    # Import CSVs
+    print("Importing CSV files...")
+    for csv_file in reports_dir.glob('**/*.csv'):
+        try:
+            import_head_rankings(csv_file, repo_path=args.repo)
+        except Exception as e:
+            print(f"✗ Failed to import {csv_file.name}: {e}")
+
+    # Import JSONs
+    print("\nImporting JSON files...")
+    for json_file in reports_dir.glob('**/*.json'):
+        # Skip manifest files
+        if 'manifest' in json_file.name.lower():
+            continue
+        try:
+            import_json_metrics(json_file, repo_path=args.repo)
+        except Exception as e:
+            print(f"✗ Failed to import {json_file.name}: {e}")
+
+    print("\n✓ Import complete! Launch UI with: aim up")
+
+if __name__ == '__main__':
+    main()
+```
+
+**Run import:**
+```bash
+python scripts/import_to_aim.py
+```
+
+---
+
+### Phase 3: Custom Visualizations (2 hours)
+
+**Goal:** Add tinyLab-specific visualizations to Aim UI
+
+**`lab/tracking/visualizations.py`:**
+```python
+"""Custom visualizations for Aim UI."""
+import matplotlib.pyplot as plt
+import numpy as np
+from typing import List, Dict, Any
+
+class MIVisualizations:
+    """Mechanistic interpretability visualizations."""
+
+    @staticmethod
+    def plot_layer_metrics(metrics_by_layer: Dict[int, Dict[str, float]], metric_name: str):
+        """
+        Plot metric evolution across layers.
+
+        Args:
+            metrics_by_layer: {layer_idx: {metric_name: value}}
+            metric_name: Which metric to plot
+
+        Returns:
+            matplotlib.Figure
+        """
+        layers = sorted(metrics_by_layer.keys())
+        values = [metrics_by_layer[l][metric_name] for l in layers]
+
+        fig, ax = plt.subplots(figsize=(10, 6))
+        ax.plot(layers, values, marker='o', linewidth=2, markersize=8)
+        ax.set_xlabel('Layer', fontsize=12)
+        ax.set_ylabel(metric_name.replace('_', ' ').title(), fontsize=12)
+        ax.set_title(f'{metric_name.replace("_", " ").title()} Across Layers', fontsize=14)
+        ax.grid(True, alpha=0.3)
+
+        return fig
+
+    @staticmethod
+    def plot_head_heatmap(head_metrics: Dict[tuple, float], n_layers: int, n_heads: int):
+        """
+        Plot heatmap of head-level metrics.
+
+        Args:
+            head_metrics: {(layer, head): metric_value}
+            n_layers: Number of layers
+            n_heads: Number of heads per layer
+
+        Returns:
+            matplotlib.Figure
+        """
+        # Create matrix
+        matrix = np.zeros((n_layers, n_heads))
+        for (layer, head), value in head_metrics.items():
+            matrix[layer, head] = value
+
+        fig, ax = plt.subplots(figsize=(12, 8))
+        im = ax.imshow(matrix, cmap='RdYlGn', aspect='auto')
+
+        ax.set_xlabel('Head', fontsize=12)
+        ax.set_ylabel('Layer', fontsize=12)
+        ax.set_title('Head Ablation Effects (ΔLD)', fontsize=14)
+
+        # Add colorbar
+        cbar = plt.colorbar(im, ax=ax)
+        cbar.set_label('Logit Difference', fontsize=12)
+
+        # Add grid
+        ax.set_xticks(np.arange(n_heads))
+        ax.set_yticks(np.arange(n_layers))
+        ax.grid(which='major', color='white', linewidth=0.5)
+
+        return fig
+
+    @staticmethod
+    def plot_emergence_curve(
+        checkpoint_metrics: Dict[int, float],
+        checkpoint_steps: List[int],
+        metric_name: str = "logit_diff"
+    ):
+        """
+        Plot metric emergence across training checkpoints (for Pythia).
+
+        Args:
+            checkpoint_metrics: {checkpoint_step: metric_value}
+            checkpoint_steps: List of checkpoint steps
+            metric_name: Metric to plot
+
+        Returns:
+            matplotlib.Figure
+        """
+        steps = sorted(checkpoint_steps)
+        values = [checkpoint_metrics.get(s, 0) for s in steps]
+
+        fig, ax = plt.subplots(figsize=(10, 6))
+        ax.plot(steps, values, marker='o', linewidth=2, markersize=8, color='#2E86AB')
+        ax.set_xlabel('Training Steps', fontsize=12)
+        ax.set_ylabel(metric_name.replace('_', ' ').title(), fontsize=12)
+        ax.set_title(f'{metric_name.replace("_", " ").title()} Emergence', fontsize=14)
+        ax.set_xscale('log')
+        ax.grid(True, alpha=0.3)
+
+        # Add shaded region for crystallization
+        if len(values) > 2:
+            # Find inflection point (simple heuristic)
+            diffs = np.diff(values)
+            inflection = np.argmax(diffs) + 1
+            ax.axvspan(steps[0], steps[inflection], alpha=0.1, color='red', label='Pre-crystallization')
+            ax.axvspan(steps[inflection], steps[-1], alpha=0.1, color='green', label='Post-crystallization')
+            ax.legend()
+
+        return fig
+
+    @staticmethod
+    def plot_ov_token_projection(
+        token_embeddings: np.ndarray,
+        token_labels: List[str],
+        title: str = "OV Circuit Token Projection"
+    ):
+        """
+        Plot 2D projection of OV-projected tokens.
+
+        Args:
+            token_embeddings: (n_tokens, embedding_dim) array
+            token_labels: List of token strings
+            title: Plot title
+
+        Returns:
+            matplotlib.Figure
+        """
+        from sklearn.decomposition import PCA
+
+        # Project to 2D
+        pca = PCA(n_components=2)
+        embeddings_2d = pca.fit_transform(token_embeddings)
+
+        fig, ax = plt.subplots(figsize=(12, 8))
+
+        # Scatter plot
+        scatter = ax.scatter(
+            embeddings_2d[:, 0],
+            embeddings_2d[:, 1],
+            c=range(len(token_labels)),
+            cmap='viridis',
+            s=100,
+            alpha=0.6
+        )
+
+        # Add labels
+        for i, label in enumerate(token_labels):
+            ax.annotate(
+                label,
+                (embeddings_2d[i, 0], embeddings_2d[i, 1]),
+                fontsize=9,
+                alpha=0.8
+            )
+
+        ax.set_xlabel(f'PC1 ({pca.explained_variance_ratio_[0]:.1%} var)', fontsize=12)
+        ax.set_ylabel(f'PC2 ({pca.explained_variance_ratio_[1]:.1%} var)', fontsize=12)
+        ax.set_title(title, fontsize=14)
+        ax.grid(True, alpha=0.3)
+
+        return fig
+```
+
+---
+
+### Phase 4: DVC Integration (30 min)
+
+**Goal:** Ensure Aim storage works with DVC
+
+**Update `.gitignore`:**
+```gitignore
+# Aim tracking (local only, regenerate from DVC data)
+/.aim/
+```
+
+**Optional:** Track Aim exports with DVC:
+
+```python
+# scripts/export_aim_reports.py
+"""Export Aim runs to static JSON for DVC tracking."""
+from aim import Repo
+import json
+
+repo = Repo('.')
+
+# Export all runs metadata
+runs_data = []
+for run in repo.iter_runs():
+    runs_data.append({
+        'hash': run.hash,
+        'name': run.name,
+        'experiment': run.experiment,
+        'creation_time': run.creation_time.isoformat(),
+        'params': run.get('hparams', {}),
+        'metrics': {
+            track.name: track.values.last_value()
+            for track in run.metrics()
+        }
+    })
+
+# Save to reports/
+with open('reports/aim_runs_export.json', 'w') as f:
+    json.dump(runs_data, f, indent=2)
+
+# Track with DVC
+# dvc add reports/aim_runs_export.json
+```
+
+---
+
+## Usage Guide
+
+### Running Experiments with Tracking
+
+**Before (no tracking):**
+```bash
+python -m lab.battery --config lab/configs/run_h1_cross_condition_balanced.json
+```
+
+**After (with Aim tracking):**
+```bash
+# Tracking is automatic! Just run as before
+python -m lab.battery --config lab/configs/run_h1_cross_condition_balanced.json
+
+# View in UI
+aim up
+```
+
+### Browsing Experiments
+
+**Launch UI:**
+```bash
+aim up
+# Opens http://localhost:43800
+```
+
+**UI Features:**
+
+1. **Runs Table** - View all runs with metadata, hyperparameters, metrics
+2. **Metrics Explorer** - Compare metrics across runs with interactive plots
+3. **Images** - Browse attention patterns, OV projections, calibration curves
+4. **Text Logs** - View model outputs, errors, notes
+5. **Params** - Filter and group by hyperparameters
+6. **Custom Dashboards** - Create saved views for specific analyses
+
+### Filtering and Grouping
+
+**In UI:**
+- Filter by model: `run.model_name == "gpt2-medium"`
+- Filter by condition: `run.condition == "facts"`
+- Group by hypothesis: Group by `run.hypothesis`
+- Compare top heads: Filter by `logit_diff > 2.0` with context `{layer: 0}`
+
+**Programmatically:**
+```python
+from aim import Repo
+
+repo = Repo('.')
+
+# Find all GPT-2 Medium facts runs
+runs = repo.query_runs(
+    "run.model_name == 'gpt2-medium' and run.condition == 'facts'"
+).iter()
+
+for run in runs:
+    print(f"Run {run.hash}: LD = {run.metrics()['logit_diff'].last_value()}")
+```
+
+### Creating Custom Dashboards
+
+**Example: Suppressor Analysis Dashboard**
+
+```python
+# In Aim UI → Metrics → Create new dashboard
+# Add charts:
+# 1. Logit Diff by Layer (line plot, group by layer)
+# 2. Head Heatmap (table, context: {layer, head})
+# 3. OV Fidelity Over Time (line plot, group by checkpoint)
+# 4. Attention Patterns (image grid, filter: layer == 0)
+
+# Save as "Suppressor Analysis"
+```
+
+---
+
+## Advanced Features
+
+### 1. Compare Runs Side-by-Side
+
+```python
+from aim import Repo
+
+repo = Repo('.')
+
+# Get two runs
+run1 = repo.get_run('abc123')  # GPT-2 Medium facts
+run2 = repo.get_run('def456')  # Mistral facts
+
+# Compare metrics
+for metric_name in ['logit_diff', 'accuracy', 'calibration_ece']:
+    val1 = run1.metrics()[metric_name].last_value()
+    val2 = run2.metrics()[metric_name].last_value()
+    print(f"{metric_name}: GPT-2={val1:.3f}, Mistral={val2:.3f}")
+```
+
+### 2. Export for Paper
+
+```python
+# Export specific metrics for LaTeX table
+from aim import Repo
+import pandas as pd
+
+repo = Repo('.')
+
+# Query runs
+runs = repo.query_runs("run.condition == 'facts'").iter()
+
+# Build dataframe
+data = []
+for run in runs:
+    data.append({
+        'Model': run['model_name'],
+        'ΔLD': run.metrics()['logit_diff'].last_value(),
+        'Accuracy': run.metrics()['accuracy'].last_value(),
+        'ECE': run.metrics()['calibration_ece'].last_value(),
+    })
+
+df = pd.DataFrame(data)
+print(df.to_latex(index=False, float_format='%.3f'))
+```
+
+### 3. Automated Analysis Pipelines
+
+```python
+# scripts/analyze_latest_run.py
+"""Analyze most recent run and generate report."""
+from aim import Repo
+
+repo = Repo('.')
+
+# Get latest run
+run = sorted(repo.iter_runs(), key=lambda r: r.creation_time, reverse=True)[0]
+
+print(f"Latest Run: {run.hash}")
+print(f"Experiment: {run.experiment}")
+print(f"Model: {run['model_name']}")
+print(f"\nTop Metrics:")
+for name in ['logit_diff', 'accuracy', 'p_drop']:
+    print(f"  {name}: {run.metrics()[name].last_value():.3f}")
+
+# Find top suppressor heads
+head_metrics = {}
+for track in run.metrics():
+    if track.name == 'logit_diff' and track.context.get('layer') == 0:
+        head = track.context['head']
+        head_metrics[head] = track.values.last_value()
+
+top_heads = sorted(head_metrics.items(), key=lambda x: x[1], reverse=True)[:5]
+print(f"\nTop 5 Suppressor Heads (L0):")
+for head, ld in top_heads:
+    print(f"  Head {head}: ΔLD = {ld:.3f}")
+```
+
+---
+
+## Migration Checklist
+
+- [ ] Install Aim: `pip install aim`
+- [ ] Create `lab/tracking/` module
+- [ ] Add `TinyLabTracker` class
+- [ ] Integrate tracking into `lab/harness.py`
+- [ ] Test with single experiment run
+- [ ] Import historical results: `python scripts/import_to_aim.py`
+- [ ] Launch UI: `aim up`
+- [ ] Verify metrics, images, distributions appear correctly
+- [ ] Create custom dashboards for key analyses
+- [ ] Add `.aim/` to `.gitignore`
+- [ ] Update documentation (DVC_SETUP.md, README.md)
+- [ ] Train team on Aim UI usage
+
+---
+
+## FAQ
+
+**Q: How does Aim compare to MLflow?**
+A: Aim has a more modern UI, better metric comparison, and is specifically designed for ML/DL experiments. MLflow is more general-purpose with deployment features we don't need.
+
+**Q: Will this slow down experiments?**
+A: Minimal overhead (<1% for typical runs). Logging is asynchronous.
+
+**Q: Can I disable tracking?**
+A: Yes, just don't initialize `TinyLabTracker`. Or use env var: `TINYLAB_DISABLE_TRACKING=1`.
+
+**Q: How much storage does Aim use?**
+A: ~1-5MB per run for metrics/metadata. Images/distributions increase this. Use `aim storage --clean` to remove old runs.
+
+**Q: Can I query Aim from notebooks?**
+A: Yes! See examples above. Full Python API available.
+
+**Q: How do I backup Aim data?**
+A: The `.aim/` directory contains everything. Can export to JSON for DVC tracking or copy entire directory.
+
+---
+
+## Next Steps
+
+1. **Implement Phase 1** - Basic tracking (30 min)
+2. **Test with one experiment** - Verify tracking works (15 min)
+3. **Import historical data** - Run import script (30 min)
+4. **Explore UI** - Familiarize with Aim interface (30 min)
+5. **Add custom visualizations** - Implement MI-specific plots (2 hours)
+6. **Create dashboards** - Build saved views for analyses (1 hour)
+7. **Document for team** - Write usage guide (1 hour)
+
+**Total time:** ~5-6 hours for complete integration
+
+---
+
+## Resources
+
+- **Aim Docs:** https://aimstack.readthedocs.io/
+- **Aim GitHub:** https://github.com/aimhubio/aim
+- **Aim Discord:** https://community.aimstack.io/
+- **Examples:** https://github.com/aimhubio/aim/tree/main/examples
+
+---
+
+**Document Version:** 1.0
+**Author:** Claude
+**Status:** Ready for Implementation
diff --git a/scripts/setup_aim.sh b/scripts/setup_aim.sh
new file mode 100755
index 0000000..29ecb3d
--- /dev/null
+++ b/scripts/setup_aim.sh
@@ -0,0 +1,74 @@
+#!/usr/bin/env bash
+#
+# setup_aim.sh - Quick setup for Aim experiment tracking
+#
+# Usage:
+#   ./scripts/setup_aim.sh
+
+set -euo pipefail
+
+# Colors
+GREEN='\033[0;32m'
+BLUE='\033[0;34m'
+YELLOW='\033[1;33m'
+NC='\033[0m'
+
+echo -e "${BLUE}╔════════════════════════════════════════╗${NC}"
+echo -e "${BLUE}║  tinyLab Aim Tracking Setup           ║${NC}"
+echo -e "${BLUE}╚════════════════════════════════════════╝${NC}"
+echo
+
+# 1. Install Aim
+echo -e "${BLUE}[1/4]${NC} Installing Aim..."
+pip install aim
+echo -e "${GREEN}✓${NC} Aim installed\n"
+
+# 2. Create tracking module
+echo -e "${BLUE}[2/4]${NC} Creating tracking module..."
+mkdir -p lab/tracking
+
+# Create __init__.py
+cat > lab/tracking/__init__.py << 'EOF'
+"""Experiment tracking with Aim."""
+from .tracker import TinyLabTracker
+
+__all__ = ['TinyLabTracker']
+EOF
+
+echo -e "${GREEN}✓${NC} Created lab/tracking/\n"
+
+# 3. Add .aim to .gitignore
+echo -e "${BLUE}[3/4]${NC} Updating .gitignore..."
+if ! grep -q "^/.aim/" .gitignore 2>/dev/null; then
+    cat >> .gitignore << 'EOF'
+
+# Aim experiment tracking (regenerate from data, don't commit)
+/.aim/
+EOF
+    echo -e "${GREEN}✓${NC} Updated .gitignore\n"
+else
+    echo -e "${YELLOW}→${NC} .aim/ already in .gitignore\n"
+fi
+
+# 4. Initialize Aim repo
+echo -e "${BLUE}[4/4]${NC} Initializing Aim repository..."
+python << 'EOF'
+from aim import Repo
+repo = Repo.from_path('.', init=True)
+print(f"Initialized Aim repo at: {repo.path}")
+EOF
+echo -e "${GREEN}✓${NC} Aim repository initialized\n"
+
+# Done!
+echo -e "${GREEN}╔════════════════════════════════════════╗${NC}"
+echo -e "${GREEN}║  Setup Complete! 🎉                    ║${NC}"
+echo -e "${GREEN}╚════════════════════════════════════════╝${NC}"
+echo
+echo "Next steps:"
+echo "  1. Copy tracker.py from AIM_INTEGRATION_PLAN.md to lab/tracking/"
+echo "  2. Import historical results: python scripts/import_to_aim.py"
+echo "  3. Launch UI: aim up"
+echo "  4. Browse experiments at http://localhost:43800"
+echo
+echo "See AIM_INTEGRATION_PLAN.md for detailed integration guide."
+echo

From b5d181effe70d1b36dd5189ee4533e503e0482ba Mon Sep 17 00:00:00 2001
From: Claude <noreply@anthropic.com>
Date: Tue, 18 Nov 2025 20:01:43 +0000
Subject: [PATCH 3/3] Integrate Stage-1A artifacts into DVC and Aim plans

Updated DVC_MIGRATION_DESIGN.md and AIM_INTEGRATION_PLAN.md to include
the new Stage-1A developmental interpretability artifacts:

**DVC Updates:**
- Added task_b_weekdays.jsonl to data tracking (~10K)
- Added Stage-1A results: circularity summaries and VDI runs (~50K)
- Updated total: ~7.5 MB across 360+ files

**Aim Updates:**
- Added note about Stage-1A developmental interpretability coverage
- Will track: VDI metrics, circularity, induction head emergence, head
  scaling factors, synchronization control metrics

**README Integration:**
- Successfully merged Stage-1A dry run instructions with DVC section
- Both sections now coexist cleanly in Quick Start

All changes rebased on top of latest main (db92c4e) which includes
Stage-1A prereg, small-model VDI/circularity tooling, head scaling
hooks, and README updates.
---
 AIM_INTEGRATION_PLAN.md |  4 ++++
 DVC_MIGRATION_DESIGN.md | 12 +++++++-----
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/AIM_INTEGRATION_PLAN.md b/AIM_INTEGRATION_PLAN.md
index fef5da7..7b49187 100644
--- a/AIM_INTEGRATION_PLAN.md
+++ b/AIM_INTEGRATION_PLAN.md
@@ -4,12 +4,16 @@
 **Date:** 2025-11-18
 **Purpose:** Add comprehensive experiment tracking with web UI for mechanistic interpretability research
 
+**Note:** This plan now includes Stage-1A developmental interpretability metrics (VDI, circularity, induction head emergence) alongside the core suppressor analysis work.
+
 ---
 
 ## Overview
 
 This document outlines the integration of [Aim](https://aimstack.io/) experiment tracking into tinyLab. Aim will provide an interactive web UI to browse experiments, compare runs, and visualize mechanistic interpretability metrics in real-time.
 
+**Coverage:** Supports both the main suppressor paper experiments AND the Stage-1A preregistered pilot on early-layer synchronization control.
+
 ### Why Aim?
 
 - **Self-hosted** - No cloud dependencies, works offline
diff --git a/DVC_MIGRATION_DESIGN.md b/DVC_MIGRATION_DESIGN.md
index 8c60042..1618914 100644
--- a/DVC_MIGRATION_DESIGN.md
+++ b/DVC_MIGRATION_DESIGN.md
@@ -17,11 +17,13 @@ This document outlines the design for migrating tinyLab to use DVC (Data Version
 | Raw Data | `lab/data/corpora/` | 18 | ~370K | JSONL datasets (facts, counterfactual, logical, negation) |
 | Data Splits | `lab/data/splits/` | 18 | ~29K | Train/val/test indices |
 | Lexicons | `data/lexicons/` | 1 | 949B | Hedge/booster word lists |
-| Results (CSV) | `reports/` | 161 | ~4.5MB | Head rankings, layer sweeps, summaries |
-| Results (JSON) | `reports/` | 137 | ~2.8MB | Metrics, analyses, manifests |
+| Stage-1A Data | `lab/data/task_b_weekdays.jsonl` | 1 | ~10K | Task-B weekday modular addition data |
+| Results (CSV) | `reports/` | 161+ | ~4.5MB | Head rankings, layer sweeps, summaries |
+| Results (JSON) | `reports/` | 137+ | ~2.8MB | Metrics, analyses, manifests |
+| Stage-1A Results | `reports/task_b_circularity_*.json`, `reports/pilot_stage1a/` | 5+ | ~50K | Circularity summaries, VDI runs |
 | Paper Supplements | `paper/supplement/` | 20+ | ~150K | Bootstrap CI, calibration, validation data |
 
-**Total data to track with DVC: ~7.4 MB across 355+ files**
+**Total data to track with DVC: ~7.5 MB across 360+ files** (including Stage-1A pilot artifacts)
 
 ### Data Already Gitignored (stays ignored)
 
@@ -96,9 +98,9 @@ tinyLab/
 
 **DVC Tracking Scheme:**
 - `data/lexicons/*.json` → Track individual files
-- `lab/data/corpora/` → Track entire directory
+- `lab/data/corpora/` → Track entire directory (includes task_b_weekdays.jsonl)
 - `lab/data/splits/` → Track entire directory
-- `reports/` → Track entire directory (includes all CSV/JSON)
+- `reports/` → Track entire directory (includes all CSV/JSON, Stage-1A circularity/VDI results)
 - `paper/supplement/` → Track entire directory
 
 ---