feat: RL infrastructure — agent environment, trajectory export (ATIF), benchmark loaders (SWE-bench / Terminal-Bench / SWE-smith / EdgeBench), and verl / TRL adapters

[Enderfga]

Turns OpenRath from an agent orchestration framework into infrastructure you can collect RL rollouts with, evaluate against public benchmarks with, and hand to a trainer.

This is a large, opinionated addition. It is offered as much as a reference implementation as a merge request — if it is out of scope for the main line, the branch stands on its own and can be used from a fork. Everything below is verifiable from the tests in this PR.

---

What you can do with it

# 1. Run an episode under an enforced permission boundary
env = OpenRathEnv(OpenRathEnvConfig(
    backend="opensandbox",
    tool_policy=ToolPolicy(fs_roots=("/workspace",), command_deny=("rm",), max_calls=200),
))

# 2. Collect rollouts in parallel and hand them to a trainer
batch = collect_benchmark_rollouts(tasks, policy, max_workers=8)
proto   = to_verl_data_proto(batch)   # verl DataProto
dataset = to_trl_dataset(batch)       # TRL / GRPO / PPO

# 3. Load a public benchmark, and be told what your backend cannot run
report = load_terminal_bench(tasks_dir, features=backend.capabilities().features)
print(report.summary())   # "229 runnable, 12 skipped (95% coverage)"
for skipped in report.skipped:
    print(skipped.task_id, skipped.reason)   # "... backend lacks: compose"

# 4. Mine forked branches for preference pairs
pairs = extract_preference_pairs(build_session_graph(sessions, rewards=rewards))

What is in it

Environment and permissions. OpenRathEnv is a transactional episode state
machine over the existing Session/Sandbox/FlowToolCall stack: reset() rolls back completely on failure, a started action keeps its trajectory record even if reward or verification then fails, and normal terminal paths close the session WAL while faults abandon it.

ToolPolicy bounds what a model under training can reach — tool allowlist, filesystem roots, command allow/deny, per-episode call budget. It is enforced in dispatch_flow_tool(), the single point every tool call passes through, so a permission cannot be talked around by the model. A denied call never reaches the tool body; it returns as a recorded ToolExecutionFailure the model can react to, rather than an exception that kills the episode.

Trajectories. A compact, versioned episode protocol (episode_start → step* → episode_end) whose size grows linearly with the transcript, rather than storing a full observation per step. to_atif() exports ATIF-v1.7, the interchange format Harbor, TRL, and SkyRL already read.

rath.data exports the Session lineage DAG — the piece ATIF cannot express. ATIF nests subagents but has no fork/merge graph; OpenRath's lineage has both. extract_preference_pairs() turns scored siblings of a common parent into chosen/rejected pairs for DPO or reward-model training: siblings saw the same state before diverging, which is the only comparison here not confounded by a different starting point.

Benchmarks. Four loaders map external datasets onto one task/verifier protocol.
Each returns a LoaderReport, never a bare task list — a score computed over a silently truncated subset is not a score:

Loader	Verifier	Requires
load_swebench (SWE-bench Verified)	FAIL_TO_PASS / PASS_TO_PASS, binary	per-task image
load_terminal_bench (Terminal-Bench)	task's own test suite	per-task image; compose for 12 of 241 tasks
load_swesmith (SWE-smith)	binary test lists	per-task image (repository-level, ~250 total)
load_edgebench (EdgeBench / SForge)	judge container	host Docker daemon

Trainer adapters. to_verl_data_proto() produces a real verl.protocol.DataProto;
to_trl_dataset() produces a datasets.Dataset for TRL. Both import lazily — import rath.training still pulls in no verl, torch, numpy, tensordict, trl, or datasets.

Decisions worth arguing with

These are the places where the obvious implementation is wrong, and the code deliberately does something else.

Network isolation is a backend capability, not a policy field. Denying curl
by name cannot stop socket.connect inside an interpreter. A bound that can be stepped over is worse than no bound, because it reads like protection. So ToolPolicy has no network flag; BackendCapability.NETWORK_ISOLATION exists instead, and a task declaring internet=False is skipped on a backend that cannot enforce it.

Loaders publish coverage, including zero. EdgeBench scores inside a second
container the harness starts, and its own documentation says running the harness inside a container hits Docker-in-Docker problems. On a container-based sandbox every EdgeBench task is therefore reported as skipped, with host_docker named as the missing capability. The loader exists for protocol compatibility; it does not claim a full EdgeBench run is possible here.

All 241 Terminal-Bench tasks ship a docker-compose.yaml, so the file's
existence says nothing about whether a task needs multi-container orchestration. Only the service count does — 229 declare one service, 12 declare more. Keying on the file would have skipped 229 runnable tasks.

Built-in tools are process-wide singletons, so serializing "unsafe" tools per
instance would make run_shell_command in one rollout wait for run_shell_command in an unrelated one. Measured: four independent sandboxes running sleep 1 concurrently took 4.68 s. Built-ins now declare sandbox_scoped = True, which narrows their lane to the calling session's sandbox — same four sandboxes now take
1.10 s, while two sessions sharing one sandbox still take turns.

ATIF is pinned at v1.7. It gained fields at 1.3, 1.4, 1.6, and 1.7; a version
bump is a code change, not a silent reinterpretation of the same document.

Exactly one DAG extractor ships. Linearizing search-and-backtrack traces and
reconstructing multi-agent provenance are both possible from the exported graph, but no consumer has asked for either, and an abstraction built for an unknown consumer is waste. They are example recipes, not APIs.

What this is not

No rollout service, no Ray, no scheduler — parallelism stays with whatever the user already runs. No Gym/Gymnasium API and no claim of Gym compatibility. No trainer tensors, tokenization, advantages, or optimization: OpenRath is the sampling layer, not the learner.

Limits, stated plainly

• No loader has been executed against a real Docker image in this branch. Field
  names and image-naming conventions were pinned against the live datasets and Docker Hub rather than written from memory, and the mapping is tested against committed fixtures — but "runs a real SWE-bench instance end to end" is not demonstrated here.
• The verl adapter is tested against a fake module implementing
  DataProto.from_dict, plus an optional CI job; a real verl install was not exercised locally.
• Credentialed suites (opensandbox, openviking, live_llm) were not run.
• The compact trajectory schema is bumped to v2 (records now carry a UTC
  created_at, which ATIF requires per step). The schema was never released, so there is no migration.

Verification

1070 passed, 10 skipped on the non-credentialed matrix; ruff check, ruff format --check, mypy src (strict) clean; uv build succeeds; examples 13–18 exit zero. 109 files, +10,818 / −254.

The 15 commits are individually reviewable and ordered by dependency: backend capabilities → per-task sandbox spec → tool policy → capability gate → trajectory timestamps → ATIF → lineage DAG → preference pairs → four loaders → TRL adapter → end-to-end and docs.

Docs

ONLINE_ENV.md covers the tool policy and why network isolation is not in it, the three trajectory layers and the ATIF version pin, and the coverage contract for every loader.