Self-Improving Loops for Visual Robotic Planning

[Project page] [arXiv] [Quick Start] [Experiments] [BibTex]

Quick Start

Install MuJoCo

Install MuJoCo if it is not already the case:

1. Download MuJoCo binary here (mujoco-py requires MuJoCo 2.1.0)
2. Unzip the downloaded archive into ~/.mujoco/
3. Add MuJoCo path ~/.mujoco/mujoco210/bin into the env variable LD_LIBRARY_PATH

Setup Conda Environment

conda create -n silvr python=3.10
conda activate silvr

pip install -r requirements.txt
pip install -e .

Pretrained Checkpoints

We provided the pretrained checkpoints for MetaWorld in-domain video model and inverse dynamics model.

SILVR Experiments

Download the pretrained checkpoints or pre-train your own in-domain video model and inverse dynamics model, and fill in load_init_ckpt_path and inv_ckpt_path with their corresponding paths in configs/default.yaml.

Example:

python silvr.py task=metaworld-door-close seed=0

Tip

To enable wandb, fill in wandb_entity and wandb_project in configs/default.yaml, and append use_wandb=True to the command above.

Note

We disabled IPA by default in MetaWorld experiments. To replicate, set inverse_probadap to false and prior_strength to 0 in configs/default.yaml

Citation

If you find this repository useful for your research, please consider citing our work:

@inproceedings{
      luo2026selfimproving,
      title={Self-Improving Loops for Visual Robotic Planning},
      author={Calvin Luo and Zilai Zeng and Mingxi Jia and Yilun Du and Chen Sun},
      booktitle={The Fourteenth International Conference on Learning Representations},
      year={2026}
    }

Acknowledgement

This repo contains code adapted from flowdiffusion, diffusion_policy, TDMPC and Adapt2Act. We thank the authors and contributors for open-sourcing their code.