Feature/rltraining grpo example - #226
Open
deepak-vij wants to merge 2 commits into
Open
Conversation
…ading Adds examples/RLTraining/train_grpo.py, a GRPO fine-tuning demo that uses AIO Sandbox as the reward environment. Each rollout writes the candidate solution and its hidden tests into the sandbox and executes them under `nono run --allow-cwd`, so every rollout is graded inside its own process-level sandbox. Completions are scored on a -1.0 to +1.0 scale based on whether the tests passed, failed, or crashed.
Brings the RLTraining example in line with the other examples: adds the uv project definition, a README covering the reward shaping and why each rollout is sandboxed, and registers it in the examples index. Also adds a Dockerfile, since this example requires the nono CLI inside the sandbox and the stock image does not ship it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title
feat(example): add GRPO RL training example with sandboxed rollout grading
Body
Summary
Adds
examples/RLTraining, an example that uses AIO Sandbox as the rewardenvironment for reinforcement learning rather than just as a code-execution
target.
It fine-tunes
Qwen/Qwen2.5-Coder-1.5B-Instructwith GRPO (TRL), where thereward comes from actually running each generated solution against hidden
tests inside the sandbox. Every rollout is graded in its own process-level
sandbox via
nono run --allow-cwd, so generated code is confined to itsworking directory and one bad generation cannot corrupt the environment that
grades the next one.
Completions are scored on a -1.0 to +1.0 scale:
+1.0ALL_TESTS_PASSED-0.5AssertionErroror runtime traceback-1.0SyntaxErroror execution failureContents
train_grpo.py— the training script and reward functionREADME.md,pyproject.toml— standard example scaffoldDockerfile— sandbox image with thenonoCLI added, since the stockimage does not ship it and the example requires it
examples/README.mdindexTesting
Ran end-to-end against a locally built sandbox image (
nono 0.61.1):+1.0, 10 ×-0.5, 8 ×-1.0Notes