RFC 006: Agentic RL through Harness Interception — token-faithful traces for TRL

[rycerzes]

Summary

Adds an RFC for training the policy driving a black-box agent harness (Pi, OpenCode, Claude Code, …) inside OpenEnv by intercepting its LLM calls at an OpenAI-compatible boundary and recording a token-faithful trace — (prompt_ids, response_ids, loss_mask, response_logprobs, reward) — that TRL consumes via rollout_func.

This turns RFC 005's CLIHarnessAdapter.run_white_box stub (currently NotImplementedError) into a real seam. It is the installed-agent training path — the counterpart to the external-agent pattern TRL's Harbor integration already covers.

Design-only RFC PR — no runtime behavior changes.

What's in it

• Ownership boundary with TRL (OpenEnv owns interception + trace; TRL owns generation, weight sync, advantages, IS correction).
• 15 design decisions with rationale/trade-offs, each backed by cross-framework evidence (NVIDIA Polar/ProRL, verifiers, AReaL, Agent Lightning, rLLM) and recent literature (TIM, SAO, TITO, loss-mask/provenance work).
• Token-level trace contract, TITO discipline via TRL's chat_template_utils, prefix-break forking, session-ID-as-API-key auth, synthetic SSE replay, weight-sync fencing, proxy-enforced budgets.
• Mermaid diagrams: architecture, prefix-break forking, request lifecycle.

Relationship to other work

• Tracking issue: Agentic RL with black-box harnesses in OpenEnv via LLM-call interception #940
• Extends RFC 005 (rfcs/005-agentic-harnesses.md) — reconciles with it, does not parallel it.
• Supersedes draft [Draft] core: harness-rollout contract + runnable example for online RL with TRL #864 (message-level contract) with the token-level contract.

Note on numbering

Filed as 006, but 006 is also used by open PRs #794 and #624 — happy to renumber to whatever's free at merge time.

cc @burtenshaw @adithya-s-k @sergiopaniego

---

Note

Low Risk
Documentation-only PR; no production code, dependencies, or behavior changes.

Overview
Adds RFC 006 (draft) and links it from rfcs/README.md under Agentic Harnesses. No runtime or API changes in this PR—design and roadmap only.

The RFC specifies how to train policies for black-box CLI harnesses (Pi, OpenCode, Claude Code, etc.) by hosting a trainer-colocated OpenAI-compatible interception server that forwards to TRL-controlled vLLM, injects return_token_ids, and records per-call TraceEntry traces (canonical prompt_token_ids, completion ids, rollout logprobs, provenance, budgets). OpenEnv stays a recorder without tokenizers or assembly; TRL HarnessRolloutWorker / _chain_to_sequences builds training rows for AsyncGRPOTrainer.

It extends RFC 005 (fills CLIHarnessAdapter.run_white_box, loop-owning wait_for_completion / fetch_proxy_trace), supersedes the message-level #864 direction, and documents 18 decisions (ownership vs TRL, trainer-side vs in-sandbox topology, session-as-API-key, prefix-based chain forking, weight-sync alignment, HF Sandbox proxy_url_for connectivity). Includes a “What OpenEnv must land” checklist and example wiring.

^{Reviewed by Cursor Bugbot for commit 0975c2d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[sergiopaniego]

linking huggingface/trl#6420 for considering it as it's directly related with this one.

[sergiopaniego]

@rycerzes solid RFC. One alignment point on the TRL seam: the harness-training path is being implemented in TRL as AsyncGRPOTrainer + HarnessRolloutWorker (huggingface/trl#6420), rather than GRPOTrainer's rollout_func as described here. I validated that branch end-to-end (OpenCode agents in remote HF sandboxes, in-sandbox proxy trace, GRPO), so it's working. Just flagging so you can take it into account and align the RFC's TRL integration section with it.

[Copilot]

Pull request overview

Adds a new design RFC describing an “installed-agent” RL training path for black-box CLI harnesses by intercepting OpenAI-compatible LLM traffic and emitting token-faithful traces consumable by TRL.

Changes:

• Introduces RFC 006 describing an interception server + trace recorder architecture for token-level rollouts.
• Defines the proposed trace contract (prompt_ids, response_ids, loss_mask, response_logprobs, reward) and key design decisions (TITO, prefix-break forking, session-as-API-key, etc.).
• Documents intended integration points with existing OpenEnv harness seams (CLIHarnessAdapter.run_white_box, build_harness_rollout_func) and TRL’s rollout_func.

Alignment Review Notes (per skill)

• Automated Checks
  • Lint: Not run (no facility to execute .claude/hooks/lint.sh in this review environment)
  • Debug code: Not run (no facility to execute .claude/hooks/check-debug.sh in this review environment)
• Open RFCs context (relevant)
  • RFC 005: In Review (rfcs/005-agentic-harnesses.md) — this RFC explicitly extends it.
  • Other RFCs in Draft/In Review exist but no direct conflict identified from this doc-only change.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[sergiopaniego]

Following up on my earlier note about the TRL seam. Now that huggingface/trl#6420 is merged, the installed-agent path in TRL is concretely AsyncGRPOTrainer + HarnessRolloutWorker (not GRPOTrainer.rollout_func / run_white_box as the RFC describes), so it would be worth aligning the integration section on that. Two more inputs from running it end to end (OpenCode + Pi, local subprocess and remote HF sandboxes):

1. Remote / managed-sandbox interception topology. The architecture assumes the harness can reach the trainer-side interception server. For agents in remote, network-isolated sandboxes (HF Sandbox, E2B) that needs spelling out: the sandbox can egress HTTPS but has no easy inbound. The shape we validated keeps the server trainer-side/colocated and exposes it to the sandbox through an egress tunnel. The harness only gets OPENAI_BASE_URL = tunnel/rollout/<sid>/v1 plus the per-rollout bearer, and nothing runs in the sandbox. Two properties. (a) Weight-sync fencing (D12) stays a local call (an in-sandbox recorder would make it distributed). (b) Less crosses the wire: the token-faithful data (ids, logprobs) is recorded trainer-side and never reaches the sandbox, only the OpenAI-dialect request and completion text do. Only the base-URL host differs between local and remote, so it generalizes cleanly. Happy to write this up as a subsection.

2. Where the TITO buffer should live. For context on the status quo: in the merged worker path, _turns_from_trace reconstructs each turn's prompt framework-side with apply_chat_template, while OpenEnv's proxy records only raw (messages, generation-time completion_token_ids, logprobs). That reconstruction is already faithful for prefix-preserving templates (vLLM re-renders per stateless call too, so both sides land on the same ids, consistent with the 18/19 you cite). TITO's real value is turning that into a guarantee and handling prefix-breaks (compaction, sub-agents) via forking, which is needed. The thing to weigh is placement. D3 puts TITO in openenv.core via trl.chat_template_utils, which couples core to a TRL internal. Since the reconstruction already lives framework-side today and OpenEnv stays a raw recorder (any trainer can consume the same trace), keeping the token-buffer on the framework side would preserve OpenEnv's framework-agnosticism, and the contract stays HarnessTrace. The counter-argument is buffer locality (verifiers/Polar assemble at the interception point), a real trade-off, just flagging it for a conscious call.

We have validated real OpenCode/Pi harnesses that could plug into the interception server, happy to help wire them up when it exists.

For reference, the in-sandbox vs trainer-side placement in a remote-sandbox run:

In-sandbox (what the merged path does today):
  [sandbox]  agent  →  in-sandbox proxy  →(tunnel)→  vLLM  [trainer host]
             the proxy records the trace inside the sandbox, trainer reads it back afterwards

Trainer-side (this RFC):
  [sandbox]  agent  →(tunnel)→  interception server  →  vLLM  [trainer host, localhost]
             the server records the trace on the trainer host, live, and owns the token buffer

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

rfcs/006-agentic-rl-harness-interception.md:7

• RFC numbering: RFC 010 explicitly states that numbers 006–009 are reserved for the self-improving-gym RFC family (rfcs/010-echo-env-token-world-model.md:8-11). Using RFC ID: 006 here (and the filename/README index) creates an in-repo inconsistency. Consider renumbering this RFC (and renaming the file + README entry), or updating RFC 010’s reservation note if that plan has changed.

**Created**: 2026-07-11
**Authors**: @rycerzes, @sergiopaniego
**RFC ID**: 006

rfcs/README.md:96

• This RFC index entry uses 006-..., but RFC 010 notes that IDs 006–009 are reserved (rfcs/010-echo-env-token-world-model.md:8-11). If that reservation still stands, this entry (and the RFC filename/header) should be renumbered to an unreserved ID to avoid conflicting documentation.

- [005-agentic-harnesses.md](./005-agentic-harnesses.md) - Agentic Harness Integration (OpenClaw, Claude Code, etc.)
- [006-agentic-rl-harness-interception.md](./006-agentic-rl-harness-interception.md) - Agentic RL through Harness Interception: token-faithful traces for TRL

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

rfcs/006-agentic-rl-harness-interception.md:6

• RFC ID 006 is currently also used by open PRs #624 and #794 (both add different rfcs/006-*.md files). Keeping this as 006 will force a renumber + file rename + README index update during merge conflict resolution; it’s safer to pick an unused RFC ID now (or mark the ID as TBD) to avoid collisions and broken cross-references.

**Created**: 2026-07-11
**Authors**: @rycerzes, @sergiopaniego
**RFC ID**: 006

[rycerzes]

@sergiopaniego Pushed a revision covering all three points.

• Seam — now AsyncGRPOTrainer + HarnessRolloutWorker; run_white_box is the secondary mode, rollout_func a sync bridge.
• Buffer — your call. My "verifiers and Polar assemble at the interception point" claim was wrong: Polar merges post-hoc in prefix_merging.py, verifiers in graph.py, TRL in _chain_to_sequences. None buffers in the proxy. OpenEnv is a recorder only.
• D3 adds one thing: inject return_token_ids and record the engine's canonical prompt_token_ids. Interstitials then come from slicing the next call's prompt after the EOT token (as Polar does) — nothing re-rendered, no tokenizer in core.
• That implies one TRL change: _turns_from_trace reading entry["prompt_token_ids"] instead of re-rendering, with a fallback. It currently passes neither chat_template nor chat_template_kwargs where _sample_turn passes both, so the training-template substitution is skipped and our enable_thinking=False injection is invisible. Divergent prompts make _chain_to_sequences FORK every turn and pair input_ids with logprobs from a different rendering.
• Topology — D18 + a section from your description, now grounded in huggingface_hub.Sandbox.proxy_url_for. Since that port proxy carries wss:// and streamed responses, the trainer can dial in to a thin in-sandbox relay rather than needing a publicly reachable address, so I wrote that up as the preferred transport with an outbound tunnel as the alternative — which of the two did you actually use? Also kept the in-sandbox proxy as a supported mode rather than legacy; it's the simpler deployment and both modes can share the recorder and TraceEntry.
• D12 fixed — the fence is TRL's WeightTransferClient.pause()/.resume() on a VLLM_SERVER_DEV_MODE=1 server, not trl vllm-serve (no pause endpoint there).

One question: D16 puts TraceEntry + the loop-owning session protocol in openenv.core.harness. Does that match your "contract stays HarnessTrace"? I made TraceEntry what OpenEnv emits and HarnessTrace the consumer-built target, since a framework-side buffer means we can't be emitting a loss_mask.

Yes please on wiring up the OpenCode/Pi harnesses once the server exists.