Simulating Infant First-Person Sensorimotor Experience via Motion Retargeting from Babies to Humanoids

This repository provides a pipeline to reconstruct infant motion from 3D keypoints by retargeting it to MIMo, a multimodal infant model, enabling the simulation of first-person sensorimotor experiences.

---

Pipeline

The processing pipeline consists of the following stages:

1. Preprocessing
   • Convert raw 3D keypoints to a standardized format
2. Resizing
   • Adapt MIMo's body proportions to match infant dimensions
3. Retargeting
   • Reconstruct the infant's motion with MIMo
4. Simulation
   • Replay and store the fist-person sensorimotor experiences

The current version of our code requires 3D keypoints, extracted either with multiview camera systems or by fitting infant priors. We are working on adding support for 2D keypoints.

---

Installation

Clone the repository:

git clone git@github.com:ctu-vras/motion-retargeting.git
cd motion-retargeting

Create a conda environment:

conda create -n retargeting python=3.12
conda activate retargeting

Install requirements:

pip install -r requirements.txt

---

Quick Start

Run the example to test the code:

bash retarget.sh data/example/config.yml

To use the retargeting pipeline on your own data, modify the config.yml file to match your data and run the script.

---

Repository Structure

motion-retargeting/
│
├── assets/
│   ├── calibration/
│   │   ├── v1.json
│   │   └── v2.json
│   │
│   ├── img/
│   │   ├── banner.png
│   │
│   └── calibration/
│       └── v1/
│           └── calibration.xml
│
├── data/
│   ├── example/
│   │   ├── __init__.py
│   │   ├── config.yml
│   │   └── example.csv
│   │
│   └── __init__.py
│
├── libs/
│   └── MIMo/
│
├── src/
│   ├── anotate/
│   │   └── contacts.py
│   ├── envs/
│   │   ├── growth.py
│   │   ├── initialize.py
│   │   ├── retarget.py
│   │   └─ simulate.py
│   │
│   ├── evaluate/
│   │   └── keypoints.py
│   │
│   ├── preprocess/
│   │   ├── to_body25.py
│   │   └── to_mimo.py
│   │
│   ├── resize/
│   │   ├── add_mocaps.py
│   │   ├── calibration.py
│   │   ├── compute.py
│   │   └── fix_mocaps.py
│   │
│   ├── retarget/
│   │   ├── keypoints.py
│   │   └── simulate.py
│   │
│   ├── __init__.py
│   └── utils.py
│
├── .gitignore
├── LICENSE
├── README.md
├── requirements.txt
└── retarget.sh

---

Citation

If you use this code, please cite:

@InProceedings{lopez2026simulating,
  author    = {Lopez, F.M. and Kanazawa, H. and Fiala, O. and Balashov, Y. and Marcel, V. and Rustler, L. and Lenz, M. and Kim, D. and Kuniyoshi, Y. and Triesch, J. and Hoffmann, M.},
  booktitle = {IEEE International Conference on Development and Learning (ICDL)},
  title     = {Simulating Infant First-Person Sensorimotor Experience via Motion Retargeting from Babies to Humanoids},
  year      = {2026},
}

Preprint available at arxiv.

---

Feel free to open an issue if you find bugs, want to make suggestions, or have any research questions.