Skip to content

feat: TDD pipeline — move adversarial_tester before builder in FeatureBench - #1319

Open
colehurwitz wants to merge 33 commits into
mainfrom
featurebench-bench
Open

feat: TDD pipeline — move adversarial_tester before builder in FeatureBench#1319
colehurwitz wants to merge 33 commits into
mainfrom
featurebench-bench

Conversation

@colehurwitz

@colehurwitz colehurwitz commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Redesign FeatureBench pipeline to use TDD pattern: researcher → strategist → adversarial_tester → builder → gate_tests → archivist
  • The adversarial_tester now writes validation tests from the spec BEFORE the builder implements code — zero cheating risk since it never sees builder output
  • On gate_tests failure, RELOOP sends back to builder only (tests written once, implementation iterates)

Key changes

  • adversarial_tester node: reads only researcher-latest.md, TDD-focused prompt (spec-only, no builder references)
  • Builder node: reads pre-written validation_tests/test_spec_compliance.py and adversarial-qa.md
  • Edges reordered: strategist → adversarial_tester → builder → gate_tests
  • gate_tests: uses docker exec {container_name} for in-container pytest execution

Test plan

  • 32/32 contributed workflow tests pass
  • Pipeline order verified: researcher → strategist → adversarial_tester → builder → gate_tests → archivist
  • adversarial_tester reads ONLY researcher-latest.md (no builder-latest.md)
  • Builder reads validation_tests and adversarial-qa.md
  • RELOOP edge: gate_tests → builder (not adversarial_tester)
  • No stale health_checker references
  • Full QA pipeline: health check PASS, code review CLEAN (7/7), adversarial test PASS (9/9)

🤖 Generated with Claude Code

colehurwitz and others added 22 commits August 12, 2026 17:21
@colehurwitz @claude
feat: upgrade featurebench to 10-node two-loop QA pipeline with agent…
35833c0 
… adapter

Upgrades the featurebench workflow from a 4-node pipeline (study → builder →
gate_verify → auto_merge) to a 10-node two-loop QA pipeline with code review
and adversarial testing before running the full test suite. Adds a FeatureBench
agent adapter for integration with the FeatureBench evaluation harness.

Workflow changes (10 nodes, 12 edges):
- QA loop: builder → code_reviewer → gate_review → adversarial_tester → gate_qa
- Test loop: health_checker → gate_tests → builder (RELOOP)
- Both loops max 3 iterations via shared builder counter
- Fully autonomous — no user gates

New files:
- factory/featurebench/agent.py — BaseAgent subclass for FeatureBench containers
- factory/featurebench/config.toml.example — config template
- tests/test_workflow_featurebench.py — 46 tests for workflow + adapter
- scripts/run_featurebench_e2e.sh — E2E integration test script

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: E2E script selects 1-2 L1 tasks via --task-id instead of full li…
7d11ee7 
…te split

The script was using --split lite which runs all ~10 tasks and wastes
API tokens. Now dynamically extracts the first 2 L1 instance_ids from
the lite split and passes them via --task-id for targeted validation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: remove problem_statement.md from featurebench researcher reads set
2a6dbe6 
The headless executor's _wait_for_reads() checks completed_files (populated
by prior nodes' writes), not the filesystem. Since problem_statement.md is
an external input written by the FeatureBench harness before the workflow
starts, no node writes it, causing the executor to time out.

The prompt_template still references the file — this only removes the
executor dependency tracking.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: remove first-pass blocking reads from featurebench builder and a…
a769f39 
…rchivist

The builder node declared reads for QA output files (code-review.md,
adversarial-qa.md, health-check.md) that only exist on RELOOP iterations.
On the first pass, _wait_for_reads() blocked indefinitely since these files
hadn't been produced yet. Similarly, the archivist declared reads for all
review files, but adversarial-qa.md is skipped when gate_review HALTs.

Fix: builder reads only .factory/strategy/current.md (its true first-pass
dependency). Archivist reads nothing (it runs last, all files exist by then).
The prompt templates still conditionally reference QA files for RELOOP context
— the agents read them if present, but the executor no longer blocks on them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: featurebench builder no-test-creation + Vertex AI auth support
b5d8505 
Two fixes for the featurebench workflow and agent adapter:

1. Builder prompt: add critical instruction not to create/recreate test
   files — the FeatureBench evaluator provides its own tests and
   overwrites anything in tests/. Also add notes to researcher and
   strategist prompts about removed test files.

2. Agent adapter: support Vertex AI auth alongside direct API key.
   - Install gcloud CLI in container setup
   - get_env_setup_script handles ANTHROPIC_API_KEY, CLAUDE_CODE_USE_VERTEX,
     ANTHROPIC_VERTEX_PROJECT_ID, CLOUD_ML_REGION, and ADC credentials
   - config.toml.example shows both auth modes
   - E2E script accepts either ANTHROPIC_API_KEY or CLAUDE_CODE_USE_VERTEX,
     reads ADC file when using Vertex

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: FeatureBench agent adapter uses claude CLI directly instead of c…
8c30537 
…loning factory repo

The adapter previously tried to git clone the private remote-factory repo
inside Docker containers, which fails without auth. Simplified to invoke
the claude CLI directly in headless mode with --allowedTools and --max-turns.

Changes:
- Remove git clone + uv sync of remote-factory from install_script
- Remove /opt/factory/.venv PATH references from install and env setup
- Remove FACTORY_RUNNER env var (no longer using factory CLI)
- Replace 'factory workflow run featurebench' with 'claude -p' in get_run_command
- Update tests to match new adapter behavior

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: remove hardcoded pytest command from FeatureBench agent prompt
f58c6d9 
The test file is deleted by the benchmark harness before the agent runs,
so this instruction was impossible to follow.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: add FeatureBench benchmark infrastructure (splits, adapter, run…
f9b53e7 
…ner, comparison)

Phase 1: Task split generator (generate_splits.py) - loads 200 tasks from
HuggingFace LiberCoders/FeatureBench, stratified-samples 40 by repo,
splits 20/10/10 train/val/test into JSONL files.

Phase 2: Update FeatureBench SkillOpt adapter with _SPLITS_DIR,
_load_split_ids() helper, split loading in setup(), seed-based offset
selection in build_train_env(), split-aware build_eval_env().

Phase 3: SKILL.md already up to date (verified via export-skills).

Phase 4: run_comparison.sh - shell script to run fb infer + fb eval
for baseline and/or factory agents on split tasks.

Phase 5: compare_results.py - reads two eval output dirs, produces
per-task comparison table, aggregate stats, and differential analysis.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz
Merge remote-tracking branch 'origin/factory/run-d646d80a' into featu…
0762b2d 
…rebench-bench
@colehurwitz @claude
feat: add FeatureBench direct-workflow benchmarking script
85fea36 
Add bench.py that extracts task repos from Docker images, runs the
factory featurebench workflow directly, and compares results against
the standard FeatureBench baseline agent. Includes CLI interface with
--factory-only, --baseline-only, --split, and --skip-eval flags.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: FeatureBench workflow agent — write problem_statement.md, fix te…
00e4cbd 
…mplate substitution, remove builder test rule

- Add prepare_run() to factory agent to write problem_statement.md into the
  container before the workflow starts (was missing entirely, causing agents
  to not know what to implement)
- Fix {project_path} template substitution in executor._run_agent() (was
  sending literal "{project_path}" strings to agents)
- Skip --dangerously-skip-permissions when running as root in containers
- Remove builder rule 5 (run tests) — health_checker handles testing,
  builder was wasting 74 min running astropy's test suite
- Add lite split files (dev/train/val/test)

Run 2 results: 1/4 resolved, 70% avg F2P (up from 0/4, 38.6% in run 1)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: FeatureBench hybrid host/container execution pipeline
fe0baa1 
Redesign FeatureBench adapter so agent orchestration runs on the HOST with
only builder and health_checker executing inside the container via podman exec.

Phase 1 - Node metadata + executor container routing:
- Add metadata dict field to Node base class in primitives.py
- Add container-aware _run_agent_in_container() and _run_fn_in_container()
  methods to WorkflowExecutor that route via podman exec
- Add context dict, pre_node_hook, and post_node_hook to executor

Phase 2 - Redesign workflow to 6-node hybrid pipeline:
- Simplify from 10-node two-loop QA pipeline to 6-node single-loop
- researcher (host) → strategist (host) → builder (container) →
  health_checker (container) → gate_tests → archivist (host, async)
- Builder and health_checker marked with metadata={'execution_context': 'container'}

Phase 3 - Adapter refactor:
- Remove Node.js/NVM/Claude Code from container install (minimal install)
- Add _sync_to_container(), _sync_from_container(), _extract_problem_statement()
  file sync methods using podman cp
- Simplify env setup (host-side execution needs fewer vars in container)

Phase 4 - Tests:
- Update tests for 6-node structure, container metadata, executor routing
- 79 tests passing across both test files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: bench.py L1/L2 task setup + agent.py output buffering fix
6624210 
Bench.py: mirror FeatureBench runtime.py initialization (masking patches,
F2P test deletion, level detection), exclude .factory/ from patches.
Agent.py: add PYTHONUNBUFFERED=1 and stdbuf -oL to fix stale infer.log.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz
Merge branch 'factory/run-b89107b2' into featurebench-bench
b4caec2 
# Conflicts:
#	factory/featurebench/agent.py
@colehurwitz @claude
fix: revert agent.py to in-container execution with buffering fix
d63afef 
The hybrid host/container architecture needs FeatureBench harness changes
to support host-side orchestration. Revert to the working in-container
approach with the 6-node pipeline and PYTHONUNBUFFERED+stdbuf buffering fix.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: hybrid host/container orchestration for FeatureBench
321d8c4 
Override run() in FactoryAgent to execute the workflow graph on the host
while routing builder/health_checker into the Docker container via
docker exec. Host nodes (researcher, strategist, archivist) use the
host's Claude Code directly; container nodes are routed by the executor's
execution_context metadata. File sync between host and container is
handled via docker cp in pre/post node hooks.

Executor changes: make container runtime (docker/podman) configurable
via context dict, support env script sourcing and conda env selection.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: resolve factory wheel via dist-info when running outside repo dir
7b4aba7 
When fb infer runs from a different working directory, factory.__file__
resolves to site-packages instead of the source repo. The dist/ directory
doesn't exist there, causing uv build to fail. Now checks
direct_url.json from the package's dist-info metadata to find the
original wheel path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: move builder to host in FeatureBench pipeline
81bbeed 
Only health_checker remains in the container (needs conda env for tests).
Builder now runs on the host for faster file I/O. After builder completes,
the full workspace is synced into the container via docker cp before
health_checker runs tests.

Pipeline: researcher (host) → strategist (host) → builder (host) →
  health_checker (container) → gate_tests (host) → archivist (host)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: multi-region Vertex AI support for concurrent FeatureBench runs
40f2c8e 
Add FACTORY_REGIONS env var to round-robin assign Vertex AI regions to
concurrent tasks, reducing API rate limit contention. Thread-safe via
extra_env on AgentRunRequest (no os.environ mutation).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: exclude .factory/ from FeatureBench patches via .gitignore
79d110b 
All git add -A calls inside the container now ensure .factory/ is in
.gitignore first, preventing workflow artifacts (reviews, strategy,
events) from polluting the patch that FeatureBench evaluates.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: replace LLM gate verdicts with deterministic pytest/spec-compli…
ae5e29f 
…ance checks

The health_checker agent was writing wrong RESOLVED verdicts:
- L2 tasks: no test files exist, agent ran smoke tests, wrote RESOLVED:true
- L1 tasks: agent dismissed failures as "pre-existing", wrote RESOLVED:true
This made the RELOOP iteration loop completely dead.

Fix: split gate into deterministic mechanisms:
- L1 (P2P tests exist): gate runs pytest directly via docker exec, exit code
  drives PROCEED/RELOOP — no agent judgment in the verdict path
- L2 (no test files): health_checker does spec-compliance validation, gate
  checks structured SPEC_COMPLIANCE: PASS/FAIL output
- Health_checker becomes diagnostics-only — never writes RESOLVED verdicts

Also adds _expand_templates() to executor for {container_name} and future
context variables in gate commands and fn node commands.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: TDD pipeline — move adversarial_tester before builder in Featur…
91f82f3 
…eBench

Redesign FeatureBench pipeline to use TDD pattern:
  researcher → strategist → adversarial_tester → builder → gate_tests → archivist

The adversarial_tester now writes validation tests from the spec BEFORE the
builder implements code. The gate_tests node runs those tests inside the
container via docker exec. On failure, the RELOOP edge sends back to builder
(not adversarial_tester), so tests are written once and implementation iterates.

Key changes:
- adversarial_tester reads only researcher-latest.md (not builder output)
- adversarial_tester prompt is TDD-focused: spec-only, no builder references
- Builder reads pre-written validation tests and adversarial-qa.md
- Edges reordered: strategist → adversarial_tester → builder → gate_tests
- 32 tests pass covering all node properties, edges, and TDD assertions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 19, 2026

Copy link
Copy Markdown

Sentrux Quality Report

Absolute

Scanning ....
[scan] git ls-files: 674 total, 662 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 662 files, 110 unique dirs, 102 cache misses, 2.7ms
[resolve] 1229 resolved, 1676 unresolved (of 2905 total specs)
[resolve_imports] project_map 2.9ms, suffix_idx 1.1ms, suffix_resolve 16.7ms, total 20.7ms
[build_graphs] 662 files | maps 1.8ms, imports 21.0ms, calls+inherit 5.6ms, total 28.4ms | 1228 import, 9889 call, 11 inherit edges
sentrux check — 3 rules checked

Quality: 4490

✗ [Error] max_cc: 3 function(s) exceed max cyclomatic complexity of 30
    factory/cli/_ceo_helpers.py:_validate_ceo_flags (cc=43)
    factory/cli/_ceo_helpers.py:_execute_ceo (cc=43)
    factory/cli/run.py:cmd_run (cc=32)

✗ 1 violation(s) found

Diff (vs base branch)

Scanning ....
[scan] git ls-files: 674 total, 662 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 662 files, 110 unique dirs, 102 cache misses, 3.0ms
[resolve] 1229 resolved, 1676 unresolved (of 2905 total specs)
[resolve_imports] project_map 3.1ms, suffix_idx 1.1ms, suffix_resolve 17.0ms, total 21.2ms
[build_graphs] 662 files | maps 2.1ms, imports 21.4ms, calls+inherit 5.9ms, total 29.4ms | 1228 import, 9889 call, 11 inherit edges
sentrux gate — structural regression check

Quality:      4492 -> 4490
Coupling:     0.79 → 0.79
Cycles:       4 → 4
God files:    3 → 3

Distance from Main Sequence: 0.34

✗ DEGRADED
  ✗ Complex functions increased: 63 → 65

colehurwitz and others added 6 commits August 19, 2026 14:08
@colehurwitz @claude
fix: gate_tests docker cp creates nested directory — rm before copy
56a7655 
docker cp creates /tmp/validation_tests/validation_tests/ when the target
directory already exists from a prior RELOOP iteration. Fix by removing
the target first and running pytest from the directory itself.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: adversarial_tester and builder scan for masked helper functions
fd41376 
The benchmark masks function bodies throughout the codebase, not just
the main interfaces. The adversarial_tester now scans for blank/stub
functions and writes tests for them. The builder now hunts for masked
helpers on the first pass, not just during RELOOP iterations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: add stub_filler node to fill masked function bodies before TDD
7ef5ee5 
The baseline outperforms TDD because it fills tiny masked helpers (Row.__init__,
XMLWriter.data, check_anyuri) in utility files the spec doesn't mention. Adding
a dedicated stub_filler node that scans ALL source files for blank function
bodies and fills them with minimal implementations before the adversarial_tester
writes tests.

Pipeline: researcher → strategist → stub_filler → adversarial_tester → builder → gate → archivist

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: remove task-specific function names from stub_filler prompt
7f13562 
Keep it generic — no astropy-specific examples like check_anyuri or data().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: stub_filler uses AST scanner to find all masked functions mecha…
14a30e2 
…nically

Replace manual file-by-file search with a Python AST walker that finds
every function with a blank/stub body across the entire codebase in
seconds. The agent then only needs to implement — not discover.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: add gate_stubs to verify all blank bodies are filled before TDD
bdb6ed2 
AST scanner runs as a deterministic gate after stub_filler. If >3 blank
function bodies remain, RELOOPs the stub_filler (max 2 iterations).
Ensures the builder starts with a fully-filled codebase.

Pipeline: researcher → strategist → stub_filler ⇄ gate_stubs → adversarial_tester → builder ⇄ gate_tests → archivist

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 19, 2026 •
edited
Loading

Copy link
Copy Markdown

⚠️ Merge conflict detected with main

The following files conflict:

  • factory/runners/bob.py
  • factory/runners/claude.py
  • factory/runners/codex.py
  • factory/runners/opencode.py
  • factory/skillopt/adapters/featurebench.py
  • factory/workflow/executor.py

Please rebase or merge main to resolve.

colehurwitz and others added 5 commits August 20, 2026 11:09
@colehurwitz @claude
feat: isolate adversarial_tester from reloop — add builder_fix node
d29ceb2 
Adversarial tester now runs one-shot only. On RELOOP, gate_tests routes
to builder_fix (not back through adversarial_tester), so the test suite
stays stable across iterations. V8 showed reloops with adversarial test
rewrites caused F2P regressions (astropy 86%→44%).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
feat: usage tracing, stub verification, and longer builder timeout
53048c9 
Three fixes for FeatureBench pipeline:
1. scan_stubs now greps for how each blank function is called elsewhere,
   showing callers' attribute names and arguments (fixes _precision_input
   vs _precision_flag naming bugs)
2. Builder prompt requires verifying ALL stubs are filled before commit
   and reporting a checklist (fixes missed files like xml/check.py)
3. Builder timeout 1800s → 3600s to match baseline think time

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: usage tracing grep pattern to match all function references
6c2675a 
The previous pattern `.{name}` only matched method calls (dot-prefixed),
missing direct function calls and attribute references. Use plain name
grep with directory exclusions and increased context (5 usages).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: tighten usage tracing grep to reduce noise
84c5b5d 
- Skip dunder methods and names < 4 chars (too generic)
- Use regex pattern matching .name or name( for precise matches
- Exclude docs/, examples/ directories
- Reduce from 5 to 3 usages per function
- Cuts output from 156KB to 78KB on Lightning workspace

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz @claude
fix: register gate writes in completed_files so downstream reads don'…
93a33fa 
…t timeout

The executor's _execute_gate method never added the gate node's writes to
completed_files. When builder_fix declared reads on gate-pytest-output.txt,
_wait_for_reads timed out after 60s even though the file existed on disk.

Also includes pre/post node hooks for gates, last-line verdict parsing,
and max_iterations exhaust → PROCEED fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@colehurwitz