|
|
|
|
A vision-language-action policy for the SO-101 arm that picks up an object and places it where a natural-language prompt tells it to, across three increasingly hard reasoning tasks.
Overview • The Three Evals • Installation • Running a Policy • Datasets and Models • Team
 |
 |
 |
| Eval 1: direct color bowl | Eval 2: compositional reasoning | Eval 3: celebrity portrait |
The same 450M policy, fine-tuned per task, running on the real SO-101 arm.
LeMonkey is our project for the Robot Learning course at ETH Zurich. The goal is one class of policy, a vision-language-action (VLA) model built on a pretrained vision-language backbone, that observes a scene through the robot camera, reads a natural-language prompt, and manipulates the object the prompt refers to. The grading rewards a policy that genuinely reasons about the prompt rather than memorizing fixed positions.
Every task uses the same backbone: SmolVLA-450M, the smallest off-the-shelf VLA, fine-tuned per task from lerobot/smolvla_base. Using the smallest capable model keeps us competitive on the course smallest-model bonus.
The project is split into three evaluations, each a harder reasoning problem on the same robot:
| Eval | Task | Prompt style | Status |
|---|---|---|---|
| Eval 1 | Banana into a colored bowl | Direct, e.g. "Put the banana in the blue bowl." | deployed |
| Eval 2 | Banana into a bowl, bowls reshuffled | Compositional, e.g. "the 2nd bowl from the left." | deployed |
| Eval 3 | Coke can onto a celebrity portrait | Identity, e.g. "Put the coke on Barack Obama." | deployed |
Each eval has its own runtime folder, README, deployed model, and dataset. This root README is the map; each eval_N/README.md is the detailed runbook.
A banana sits in a fixed position. Three colored bowls (blue, red, green) sit in fixed positions. The policy places the banana in the bowl named by the prompt.
"Put the banana in the blue colored bowl."
- Deployed model:
HBOrtiz/so101_smolvla_eval1 - Trained on:
HBOrtiz/so101_eval1, 153 teleop episodes (behavior-cloning demos plus HG-DAgger corrections) - Runbook:
eval_1/README.md
The banana stays put, but the bowls are reshuffled across positions and the prompt no longer names a color directly. The policy has to work out which bowl is meant.
"Put the banana into the 2nd bowl from the left." "Put the banana into the bowl that is not green and not blue."
- Deployed model:
HBOrtiz/so101_smolvla_eval2 - Trained on:
HBOrtiz/so101_eval2, 180 teleop episodes balanced over 6 bowl arrangements and 6 compositional prompt families - Runbook:
eval_2/README.md
Three printed celebrity portraits are laid out on the workspace. The policy places a Coke can on the portrait of the person named in the prompt, including, in the hardest tier, celebrities never seen in training.
"Put the coke on Barack Obama."
The course rule: no separate face-recognition model or external VLM may run at inference, so the deployed VLA has to do the identity reasoning itself. We solve this with co-training, training the policy jointly on robot manipulation episodes and on a vision-language grounding dataset, so celebrity knowledge ends up inside the policy weights.
- Deployed model (in-distribution celebrities):
HBOrtiz/so101_smolvla_eval3_cotrain, trained onHBOrtiz/so101_eval3_cotrain(robot episodes) plusHBOrtiz/so101_eval3_cotrain_grounding(grounding pairs) - Deployed model (broad celebrities):
HBOrtiz/so101_smolvla_eval3_broad, co-trained onHBOrtiz/so101_eval3_broad(192-celebrity robot data) plusHBOrtiz/so101_eval3_broad_grounding(192-celebrity grounding pairs) - Runbook:
eval_3/README.md
Everything runs in one conda environment, lemonkey (Python 3.12, PyTorch CUDA 12.8, lerobot==0.5.1[smolvla]).
git clone https://github.com/Ace3Z/LeMonkey.git
cd LeMonkey
bash eval_1/scripts/brev_setup.sh # canonical env recipe: miniconda, lerobot 0.5.1, ffmpeg
conda activate lemonkeybrev_setup.sh is written for a fresh NVIDIA Brev training VM, but the package set is the same on a laptop. On a consumer GPU also install: feetech-servo-sdk, rerun-sdk==0.26.2, peft==0.19.1, and opencv-python (not the -headless build).
All models and datasets live under the HBOrtiz/ organization.
hf auth login # paste a read token (write token if you will push)- SO-101 arm: a udev rule pinning the follower to
/dev/so101-follower(and/dev/so101-leaderfor the teleop arm). The user must be in thedialoutgroup. - Camera: a USB wrist camera at
/dev/video0, 640x480 at 30 fps. - Calibration: per-arm calibration JSONs are checked in under
calibration/. Symlink them into the LeRobot cache:mkdir -p ~/.cache/huggingface/lerobot ln -s "$PWD/calibration" ~/.cache/huggingface/lerobot/calibration
Each eval ships interactive rollout scripts. They download the deployed checkpoint from the Hub on first use, capture the arm home pose, run the policy for one episode against a typed prompt, and drive the arm home for the next take.
conda activate lemonkey
# Top-level wrappers (download checkpoints from HF on first use,
# or use a local copy if shipped under policy/<repo_name>/).
./run_eval_1.sh # Eval 1: direct color pick and place
./run_eval_2.sh # Eval 2: compositional instruction following
./run_eval_3.sh # Eval 3: in-distribution celebrities (default)
./run_eval_3.sh --broad # Eval 3: broad / out-of-distribution celebritiesType the prompt at the menu (for example Put the coke on Barack Obama.), watch the rollout, then type the next one or q to quit. Each eval_N/README.md documents the per-eval scripts, checkpoints, and structured-evaluation tooling.
Every trained policy and every dataset is published under the HBOrtiz/ organization on the Hugging Face Hub. The full inventory, what each artifact is and how it was built, is in DATASETS_AND_MODELS.md.
Deployed models at a glance:
| Eval | Model | Backbone | Trained on |
|---|---|---|---|
| 1 | so101_smolvla_eval1 |
SmolVLA-450M | so101_eval1 (153 ep) |
| 2 | so101_smolvla_eval2 |
SmolVLA-450M | so101_eval2 (180 ep) |
| 3 | so101_smolvla_eval3_cotrain |
SmolVLA-450M | so101_eval3_cotrain plus so101_eval3_cotrain_grounding |
| 3 | so101_smolvla_eval3_broad |
SmolVLA-450M | so101_eval3_broad (192 celebrities) |
LeMonkey/
├── README.md this file (the map)
├── DATASETS_AND_MODELS.md Hugging Face dataset and model inventory
├── run_eval_1.sh top-level Eval 1 rollout launcher
├── run_eval_2.sh top-level Eval 2 rollout launcher
├── run_eval_3.sh top-level Eval 3 rollout launcher (--broad for OOD celebrities)
├── eval_1/ Eval 1: runtime, scripts, README
├── eval_2/ Eval 2: runtime, scripts, README
├── eval_3/ Eval 3: runtime, scripts, README
│ ├── aug/ data-augmentation pipeline (celebrity portraits)
│ ├── scripts/ training, rollout, and dataset-build scripts
│ │ ├── brev/ Brev-VM training entrypoints (all four recipes)
│ │ ├── smolvla_cotrain/ SmolVLA co-training trainer (deployed)
│ │ ├── pi05_vl_cotrain/ Pi0.5 + VL bbox-grounded VQA cotrain (published variant)
│ │ └── warmstart/ PaliGemma VQA warm-start (init for Pi0.5)
│ └── tools/ dataset-verification tooling
├── calibration/ per-arm SO-101 calibration JSONs
├── media/ logos (figures/) and demo GIFs (gifs/)
└── third_party/lerobot/ LeRobot framework as a git submodule
Per-eval train/, rollouts/, evals/, and state/ folders stay local (checkpoints, recordings, and session state are not committed).
- Robot: SO-101 6-DOF arm (follower plus leader for teleop), USB wrist camera at 640x480, 30 fps.
- Inference: any NVIDIA GPU with at least 6 GB of VRAM. A laptop GPU is enough, since SmolVLA-450M is small.
- Training: an NVIDIA Brev H100 or RTX PRO 6000, or a local RTX 5090. A 25k to 45k step run takes a few hours.
Built for the Robot Learning course at ETH Zurich.
- Roham Z. Nobari (LinkedIn)
- Mahbod Tajdini (LinkedIn)
- Darius Foodeei (LinkedIn)
- Sejohn Uruthiralingam (LinkedIn)
- Hans Baumann-Ortiz (LinkedIn)
- LeRobot: robot-learning framework, dataset format, and
lerobot-record - SmolVLA: the 450M VLA backbone we fine-tune
- NVIDIA Brev: GPU compute for training
- The ETH Zurich Robot Learning course staff