Workflow Arena

Workflow Arena is a reproducible Windows/PowerShell benchmark harness for controlled comparisons of coding-agent workflows. It evaluates a bare baseline alongside Superpowers, OpenSpec, Grill Me, Ponytail, and MatrixSpec on pinned Go, JavaScript, Python, and Rust tasks. Manifests define the inputs and experimental arms; lifecycle scripts prepare isolated candidates, run agents, freeze evidence, audit workflow adoption, judge blindly, and summarize score, time, tokens, and dollar cost.

Reproducibility Status

All active inputs have a reproducible acquisition path:

• eight upstream task commits and six capability commits are publicly available on GitHub or GitCode;
• task baselines are isolated depth-one checkouts of pinned public upstream commits;
• Ground Truth references are fetched from pinned public commits and verified by Git tree hash;
• Python environments are rebuilt from pack-local version locks;
• ESLint dependencies are installed with the recorded Node.js and npm versions;
• legacy campaign paths are reconstructed from reproduction-layout.json.

A clone becomes runnable through scripts/bootstrap-inputs.ps1; no source repositories or Git trees are redistributed. MatrixSpec is fetched from its public GitCode repository. Historical results produced with the earlier local profiled snapshot remain under archive/historical-matrixspec/ and are not attributed to the new public pin.

Reproduction is protocol-level, not byte-for-byte model output. Hosted model sampling and service changes can produce different patches or scores even when inputs, model configuration, repetitions, tests, and rubric match.

Repository Layout

• engine/: manifest validation, run planning, capability installation, and lifecycle dispatch.
• packs/tasks/: portable task prompts, Ground Truth text, hooks, runtime locks, and verification rules.
• packs/capabilities/: pinned workflow-pack manifests and offline installers.
• experiments/: arms, repetitions, models, and campaign paths.
• campaigns/: publishable protocol capsules and compact reports; generated fixtures, runs, state, judge packages, and oracle checkouts are ignored.
• reports/: aggregate score, time, token, and dollar-cost outputs.
• archive/: historical evidence that is informative but not externally reproducible.
• scripts/: bootstrap, publish preflight, and historical reporting utilities.

See engine/README.md for the lifecycle and protocol.md for the original preregistered CLI experiment.

Requirements

Use Windows with PowerShell 7. Install Git, an authenticated Codex CLI, Node.js with Corepack, and uv. Install Go and Rust when running their task packs.

Real runs copy $env:USERPROFILE\.codex\auth.json into ignored per-actor homes. Acquisition may use the network. Candidate worktrees have one visible commit, no Git remotes, and no intended network access during execution.

Bootstrap a Clone

Acquire task sources, Ground Truth references, and capability checkouts at their public manifest pins. Each isolated baseline is then synchronized from its already-pinned local source checkout; baseline creation never fetches the repository from a public remote a second time:

.\scripts\bootstrap-inputs.ps1 -Scope All

The command verifies commit and tree identities, creates isolated baselines, recreates legacy junctions, builds OpenSpec, installs task dependencies, and prefetches Cargo and Go dependencies. Use -SkipRuntimeSetup -SkipTaskDependencyFetch -SkipOpenSpecBuild when only Git inputs are needed. Source/reference acquisition and dependency setup may use the network; baseline creation and per-run preparation use local Git repositories only.

Verify the local-only baseline boundary without downloading anything:

.\scripts\bootstrap-inputs.ps1 -Scope LocalBaselines
.\scripts\baseline-sync-selftest.ps1

The independent OpenCode lane provides the same no-acquisition path through .\opencode\scripts\bootstrap.ps1 -LocalBaselinesOnly.

Validate Without Spending Tokens

pwsh -NoProfile -File .\engine\selftest.ps1

$exp = '.\experiments\cli-fields-superpowers\experiment.json'
.\engine\invoke-experiment.ps1 -Experiment $exp -Action Validate
.\engine\invoke-experiment.ps1 -Experiment $exp -Action All -DryRun

Validate now checks clean pinned sources, baseline isolation, Ground Truth tree identity, runtime presence, assets, installers, and drivers. -DryRun expands Prepare → Run → Freeze → Audit → Judge → Summarize without invoking a model or spending tokens.

Run an Experiment

.\engine\invoke-experiment.ps1 -Experiment $exp -Action All

Review arm count, repetitions, model, timeouts, and expected API cost first. A campaign invokes candidate, operator, reviewer, and judge actors and can consume substantial time and quota.

The independent OpenCode/GLM agent lane lives under opencode/. It has its own runner, experiments, state, campaigns, and reports; OpenCode results are not Codex arms and must not be merged into the Codex aggregate tables.

Publishing Safely

The root .gitignore excludes external checkouts, credentials, raw sessions, worktrees, frozen fixtures, runtime state, judge packages, invalid baselines, and release binaries. Commit the harness, manifests, task packs, campaign protocol scripts/prompts, documentation, and reviewed reports. Never force-add auth.json, *.private.json, .judge-homes, codex-homes, state/, runs/, candidates/, or tools/.

Ground Truth contracts, rubrics, public-upstream oracle diffs, and custom evaluator tests are intentionally published for auditability and reproduction. Lifecycle drivers do not copy them into candidate worktrees. Accordingly, Workflow Arena is a transparent workflow benchmark, not a secrecy-based held-out benchmark; do not describe these evaluator assets as undisclosed.

Run scripts/publish-preflight.ps1 immediately before the first commit and again before pushing. It checks required ignore boundaries, tracked-secret names when Git is initialized, and machine-specific absolute paths.