chore(examples): agentic profiles for the reward-curve verification script - #5
chore(examples): agentic profiles for the reward-curve verification script#5celve wants to merge 1 commit into
Conversation
…cript
The script's W&B project is unirl-agentic-main-verification but none of its five
profiles ran an agentic recipe, so the agentic paths had no reward-curve gate.
Adds agentic-alfworld{,-partial,-async} over the existing alfworld recipes, and
extracts ar-drpo's CUDA 13 toolkit/compat block into setup_sglang_cuda() rather
than triplicating it — verified a pure move (diff of the extracted body against
the original block is empty). The toolchain version gate now covers every SGLang
profile instead of ar-drpo alone.
Test plan: bash -n passes; every ENTRY module and EXPERIMENT recipe resolves on
disk. The profiles themselves need a pod to run.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 57b0aeeb5e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| setup_sglang_cuda | ||
| export QWEN3_INSTRUCT_PATH="${QWEN3_INSTRUCT_PATH:-${REPO_ROOT}/models/local/Qwen3-4B-Instruct}" | ||
| export DATA_PATH="${DATA_PATH:-${REPO_ROOT}/data/alfworld/train.jsonl}" | ||
| export ALFWORLD_CONFIG="${ALFWORLD_CONFIG:-}" |
There was a problem hiding this comment.
When ALFWORLD_CONFIG is not already exported—the documented optional/default invocation—this assigns an empty string to every new agentic profile. All three recipes pass that value to AlfworldEnv, whose _ensure_backend() rejects an empty or nonexistent config and raises FileNotFoundError on the first rollout reset (unirl/rollout/env/alfworld.py:172-177), so none of the new profiles can run with their defaults. Resolve a readable base_config.yaml here or require and validate the variable before launching.
Useful? React with 👍 / 👎.
|
Closing: opened against the fork by mistake. Re-opening against Tencent-Hunyuan/UniRL, with the @addressed primitive and the manager move combined into one PR. |
There was a problem hiding this comment.
🟡 Not ready to approve
The new single-node agentic SGLang profiles lack required environment validation and may miss the CUDA runtime linker shim needed for TVM-FFI JIT linking.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
Adds Taiji reward-curve verification profiles for ALFWorld agentic training so the unirl-agentic-main-verification W&B project exercises agentic rollout paths, and refactors the repeated CUDA 13 + forward-compat setup into a shared helper for SGLang-based profiles.
Changes:
- Add three new verification profiles:
agentic-alfworld,agentic-alfworld-partial,agentic-alfworld-async. - Extract the repeated CUDA 13 / forward-compat setup into
setup_sglang_cuda()and apply it toar-drpoand the new profiles. - Generalize the version/toolchain gate to apply to all SGLang profiles rather than only
ar-drpo.
File summaries
| File | Description |
|---|---|
| examples/run_reward_curve_verification_taiji.sh | Adds agentic ALFWorld profiles and refactors SGLang CUDA setup/version gating for reward-curve verification runs on Taiji. |
Review details
- Files reviewed: 1/1 changed files
- Comments generated: 2
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
| export QWEN3_INSTRUCT_PATH="${QWEN3_INSTRUCT_PATH:-${REPO_ROOT}/models/local/Qwen3-4B-Instruct}" | ||
| export DATA_PATH="${DATA_PATH:-${REPO_ROOT}/data/alfworld/train.jsonl}" | ||
| export ALFWORLD_CONFIG="${ALFWORLD_CONFIG:-}" | ||
| require_file "${QWEN3_INSTRUCT_PATH}/config.json" | ||
| require_file "${DATA_PATH}" | ||
| } |
| # NVIDIA's pip toolkit has lib/libcudart.so.13 but no conventional | ||
| # lib64/libcudart.so linker name. The multinode launcher creates this | ||
| # small per-node shim before Ray starts, so SGLang TVM-FFI JIT links. | ||
| export CUDA_RUNTIME_LIB_DIR | ||
| export CUDA_RUNTIME_LINK_DIR="${CUDA_RUNTIME_LINK_DIR:-/tmp/unirl-cuda-runtime-${UID}}" | ||
| CUDA_COMPAT_DIR="${CUDA_COMPAT_DIR:-}" |
Summary
The reward-curve verification script publishes to a W&B project called
unirl-agentic-main-verification, but none of its five profiles ran an agenticrecipe — so the agentic rollout paths had no reward-curve gate at all.
Adds
agentic-alfworld,agentic-alfworld-partialandagentic-alfworld-asyncover the existing
alfworld/alfworld_grpo{,_partial,_async}recipes, unchanged.The three profiles need the same CUDA 13 toolkit and forward-compat setup that
ar-drpoalready does for SGLang's TVM-FFI JIT, so that block moves into asetup_sglang_cuda()function rather than being copied four times. The toolchainversion gate now keys off "is this an SGLang profile" instead of naming
ar-drpo.Motivation: this is the baseline harness for LIN-693 (splitting the agentic
rollout engine from the driver-side manager). A "no behaviour change" claim needs
a before-and-after reward curve, and today there is nothing to run.
Related Issue
LIN-693 (prerequisite — establishes the baseline the refactor is measured against).
Test Plan
bash -n examples/run_reward_curve_verification_taiji.sh— passes.ENTRYmodule (unirl/train_agentic_env{,_partial,_async}.py)and
EXPERIMENTrecipe (examples/alfworld/alfworld_grpo{,_partial,_async}.yaml)resolves on disk.
setup_sglang_cuda()extraction is a pure move: diffing thefunction body against the original inline block (dedented) is empty.
Not run: the profiles themselves. They need an allocated Taiji pod with the
pinned
.venv-sglang, model and ALFWorld data — unavailable in the environmentthis was authored in.
Compatibility / Risk
No behaviour change for the five existing profiles.
ar-drpoexecutes the samesetup via a function instead of inline; the version gate now also applies to the
new agentic profiles.
One thing a reviewer with pod access should confirm: I assumed the agentic
profiles need the full CUDA 13 JIT setup because their recipes use SGLang
backend: native. If they do not,setup_sglang_cudais harmless there but therequire_dist_versionpins may be stricter than necessary.Reviewer Notes
Prepared with AI assistance; a human should review the diff and run at least one
profile before merging. Checked for overlapping open PRs — there are none.
Default paths follow the existing convention (
models/local/Qwen3-4B-Instruct,data/alfworld/train.jsonl) and may need adjusting to the pod layout.Checklist