Skip to content

feat(swe): SWE-bench evaluation with AGS sandbox - #252

Merged
Lightblues merged 8 commits into
mainfrom
swe/ags
Mar 9, 2026
Merged

feat(swe): SWE-bench evaluation with AGS sandbox#252
Lightblues merged 8 commits into
mainfrom
swe/ags

Conversation

@Lightblues

Copy link
Copy Markdown
Collaborator

Summary

  • Add SWERexEnv for remote coding in AGS sandboxes with SWE-ReX runtime (bash, read_file, write_file, edit_file tools)
  • Implement SWEBenchmark with per-instance Docker image lifecycle and patch extraction
  • Add data import/export scripts and evaluation config for SWE-bench Verified/Lite/Full

Changes

  • utu/env/swerex_env.py: AGS sandbox environment with pager-disable fix
  • utu/eval/benchmarks/swe_benchmark.py: Per-instance rollout with deep-copy config, git diff via file API
  • utu/eval/processer/swe_bench.py: Preprocessor with prompt template, patch_rate metrics
  • scripts/data/process_swe_bench.py: HuggingFace → DB import with image name mapping
  • scripts/data/export_swe_bench_patches.py: DB → JSONL export for offline evaluation
  • configs/: env, agent, eval YAML configs
  • tests/env/test_swerex_env.py: 25 unit tests with mocked runtime
  • examples/swe/swe-bench.md: Usage documentation

Closes #251

@Lightblues
Lightblues merged commit bef6ba5 into main Mar 9, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: SWE-bench evaluation integration | easons_agent

1 participant