AnyDexRetarget

中文 | English

AnyDexRetarget

High-precision hand pose retargeting system. Supports two optimizers (Adaptive and KeyVector), multiple dexterous hands, and multiple hand-tracking input sources for simulation and teleoperation.

Demo

Simulation Retargeting

sim_retarget_4x2.mp4

Quest 3 Hand-Arm Teleoperation

quest_hand_arm.mp4

Features

11 Robot Hands: Shadow, Wuji, Allegro, Inspire, Ability, Leap, SVH, LinkerHand, ROHand, Unitree Dex5, Sharpa
Two Optimizers: adaptive (pinch-aware, default) and vector (key-vector matching)
High-Precision Pinch: Adaptive optimization for accurate finger-to-thumb contact
Real-time Performance: Analytical gradients + NLopt SLSQP (~2ms per frame)
Multiple Input Sources: Apple Vision Pro, Meta Quest 3, Noitom PNS-G gloves, laptop camera (MediaPipe), recorded data replay

Supported Robots

Config files are organized by optimizer type and input source:

example/config/
├── adaptive/          # AdaptiveOptimizerAnalytical (default)
│   ├── mediapipe/     # camera / video / replay input
│   ├── avp/           # Apple Vision Pro input
│   ├── quest3/        # Meta Quest 3 input
│   └── noitom/        # Noitom PNS-G gloves
└── vector/            # KeyVectorOptimizer
    ├── mediapipe/
    ├── avp/
    ├── quest3/
    └── noitom/

Robot	`--robot` value	Config suffix	Description
Shadow Hand	`shadow`	`shadow_hand`	Shadow Hand with MuJoCo Menagerie meshes (default sim target)
Wuji Hand	`wuji`	`wuji_hand`	Wuji Hand, 5 fingers / 20 DOF
Allegro Hand	`allegro`	`allegro_hand`	Allegro Hand, 4 fingers / 16 DOF
Inspire Hand	`inspire`	`inspire_hand`	Inspire Hand with mimic joints
Ability Hand	`ability`	`ability_hand`	Ability Hand with mimic joints
Leap Hand	`leap`	`leap_hand`	Leap Hand, 4 fingers / 16 DOF
SVH Hand	`svh`	`svh_hand`	Schunk SVH Hand with mimic joints
LinkerHand L21	`linkerhand_l21`	`linkerhand_l21`	LinkerHand L21
ROHand	`rohand`	`rohand`	ROHand
Unitree Dex5	`unitree_dex5`	`unitree_dex5_hand`	Unitree Dex5
Sharpa Hand	`sharpa`	`sharpa_hand`	Sharpa Wave Hand, 5 fingers / 22 DOF

Note on Noitom configs: Only shadow_hand, wuji_hand, and inspire_hand have been roughly calibrated for Noitom input. If you need to fine-tune the mapping accuracy between your hand and the robot hand, run debug_skeleton.py to visualize three skeletons side-by-side: Blue = raw input, Green = after scaling, Red = retargeted FK result. Compare the skeleton sizes and adjust the corresponding YAML config parameters (scaling, segment_scaling, key_vectors[].scale, etc.) accordingly.
cd example
python test/debug_skeleton.py --robot inspire --input noitom --noitom-local-ip 192.168.5.25

Repository Structure

├── anydexretarget/
│   ├── retarget.py                        # High-level unified interface
│   ├── robot.py                           # Pinocchio robot wrapper
│   ├── mediapipe.py                       # MediaPipe coordinate transforms
│   └── optimizer/                         # Optimizer implementations
│       ├── base_optimizer.py              # Base optimizer with FK/Jacobian
│       ├── analytical_optimizer.py        # AdaptiveOptimizerAnalytical
│       ├── key_vector_optimizer.py        # KeyVectorOptimizer
│       ├── robot_configs.py               # Robot link/URDF configurations
│       └── utils.py                       # TimingStats, LPFilter, Huber loss
├── example/
│   ├── teleop_sim.py                      # MuJoCo simulation demo
│   ├── teleop_real.py                     # Real hardware control
│   ├── input/                             # Input device modules
│   │   ├── landmark_utils.py              # Shared MediaPipe landmark processing
│   │   ├── camera.py / video.py / ...     # Input devices
│   │   └── noitom.py                      # Noitom PNS-G glove input
│   ├── test/                              # Debug & visualization tools
│   │   ├── debug_skeleton.py              # 3-skeleton comparison viewer
│   │   └── calibrate_scaling.py            # Universal segment_scaling calibration
│   ├── config/
│   │   ├── adaptive/                      # AdaptiveOptimizerAnalytical configs
│   │   │   ├── avp/                       # Apple Vision Pro
│   │   │   ├── quest3/                    # Meta Quest 3
│   │   │   ├── mediapipe/                 # Camera / video / replay
│   │   │   └── noitom/                    # Noitom PNS-G gloves
│   │   └── vector/                        # KeyVectorOptimizer configs
│   │       ├── avp/
│   │       ├── quest3/
│   │       ├── mediapipe/
│   │       └── noitom/
│   └── data/                              # Sample recordings
├── assets/                                # Robot URDF / MuJoCo assets
└── requirements.txt

Installation

Prerequisites

Python >= 3.10
(Optional) Apple Vision Pro with Tracking Streamer app
(Optional) Meta Quest 3 with Hand Tracking Streamer app
(Optional) Noitom PNS-G gloves with Axis Studio (Windows)

Install

git clone https://gitee.com/gx_robot/AnyDexRetarget.git
cd AnyDexRetarget

# Install pinocchio via conda (recommended, pre-built binaries)
conda install -c conda-forge pinocchio

# Install other dependencies
pip install -r requirements.txt
pip install -e .

Troubleshooting

macOS MuJoCo: Use mjpython instead of python:

mjpython example/teleop_sim.py --video example/data/right.mp4

Quick Start

The repository currently includes:

example/data/right.mp4: sample input video
example/data/avp1.pkl: optional recorded hand-tracking replay

Simulation

cd example

# Run the included sample video (adaptive optimizer, default)
python teleop_sim.py --video data/right.mp4 --robot shadow --hand right

# Switch to KeyVector optimizer
python teleop_sim.py --video data/right.mp4 --robot shadow --hand right --optimizer vector

# Replay the optional sample recording
python teleop_sim.py --play data/avp1.pkl --robot shadow --hand right

# Real-time with laptop camera (MediaPipe)
python teleop_sim.py --input camera --robot shadow --hand right

# Real-time with Vision Pro
python teleop_sim.py --input visionpro --robot shadow --ip <vision-pro-ip> --hand right

# Real-time with Quest 3 (via Hand Tracking Streamer)
python teleop_sim.py --input quest3 --robot shadow --port 9000 --hand right

# Real-time with RealSense
python teleop_sim.py --realsense --robot shadow --hand right --show-video

# Noitom PNS-G gloves
python teleop_sim.py --input noitom --robot inspire --hand right --noitom-local-ip 192.168.5.25

# Replay your own recording (.pkl)
python teleop_sim.py --play path/to/record.pkl --robot shadow --hand right

Real Hardware

teleop_real.py demonstrates real hardware teleoperation using Wuji Hand as an example. It sends 5 x 4 joint targets through wujihandpy. You can adapt the control loop for other robot hands.

cd example

# Live Vision Pro -> Wuji Hand (adaptive)
python teleop_real.py --robot wuji --input visionpro --ip <vision-pro-ip> --hand right

# Live Vision Pro -> Wuji Hand (vector optimizer)
python teleop_real.py --robot wuji --input visionpro --ip <vision-pro-ip> --hand right --optimizer vector

# Noitom PNS-G gloves -> Inspire Hand
python teleop_real.py --robot inspire --input noitom --hand right --noitom-local-ip 192.168.5.25

# Replay the optional sample recording -> Wuji Hand
python teleop_real.py --robot wuji --play data/avp1.pkl --hand right

# Linux USB permission
sudo chmod a+rw /dev/ttyUSB0

Command Reference

Input Source

Option	Default	Description
`--input`	-	`teleop_sim.py`: `visionpro` / `quest3` / `noitom` / `camera` / `realsense` / `video` / `mediapipe_replay`
`--input`	-	`teleop_real.py`: `visionpro` / `noitom` / `mediapipe_replay`
`--hand`	`right`	Hand side (`left`/`right`)
`--realsense`	off	Shortcut for `--input realsense`
`--play FILE`	-	Replay recording (shortcut for `--input mediapipe_replay`)
`--video FILE`	-	Video file input with MediaPipe hand detection
`--ip`	`192.168.50.127`	Vision Pro IP
`--port`	`9000`	Quest 3 HTS listener port
`--protocol`	`udp`	Quest 3 HTS transport protocol (`udp`/`tcp`)
`--noitom-local-ip`	`192.168.5.25`	Noitom: local IP (this machine)
`--noitom-local-port`	`8000`	Noitom: local UDP port
`--noitom-server-ip`	`192.168.5.33`	Noitom: Axis Studio IP (Windows)
`--noitom-server-port`	`9000`	Noitom: Axis Studio port

Optimizer

Option	Default	Description
`--optimizer`	`adaptive`	Optimizer type: `adaptive` or `vector`
`--config`	auto-select	Configuration file (overrides `--robot` and `--optimizer`)

Robot Hand & Output

Option	Default	Description
`--robot`	`shadow` (sim) / `wuji` (real)	Robot hand type
`--record`	-	Record input data
`--output FILE`	-	Output file path for recording
`--show-video`	off	Show RGB / landmark preview for supported inputs
`--speed`	`1.0`	Playback speed
`--no-loop`	-	Disable looping for replay
`--headless`	off	Run simulation without GUI viewer
`--save-sim FILE`	-	Save offscreen simulation video
`--save-qpos FILE`	-	Save target / simulated qpos trajectory

Debug & Visualization Tools

debug_skeleton.py

Compare three hand skeletons in the MuJoCo viewer to debug retargeting issues:

Blue: Raw MediaPipe skeleton (after coordinate transform, before scaling)
Green: Scaled target skeleton (what the optimizer tries to match)
Red: Robot FK skeleton (retargeting result)

cd example

# With camera input
python test/debug_skeleton.py --robot leap --input camera

# With video file
python test/debug_skeleton.py --robot leap --video data/right.mp4

# With optional sample recording, compare optimizers
python test/debug_skeleton.py --robot shadow --play data/avp1.pkl --optimizer adaptive
python test/debug_skeleton.py --robot shadow --play data/avp1.pkl --optimizer vector

# With Noitom PNS-G gloves
python test/debug_skeleton.py --robot inspire --input noitom --noitom-local-ip 192.168.5.25

# With Noitom + KeyVector optimizer
python test/debug_skeleton.py --robot inspire --input noitom --optimizer vector --noitom-local-ip 192.168.5.25

# With RealSense D435
python test/debug_skeleton.py --robot sharpa --input realsense --hand right

# With Vision Pro
python test/debug_skeleton.py --robot sharpa --input avp --avp-ip 192.168.5.32 --hand right

# With your own recorded data
python test/debug_skeleton.py --robot shadow --play path/to/record.pkl

calibrate_scaling.py

Calibrate segment_scaling for any robot hand and input source. Collects data while the user holds their hand flat, then computes the ratio between robot FK and human bone distances.

cd example

# Calibrate with RealSense
python test/calibrate_scaling.py --robot sharpa --input mediapipe

# Calibrate with video
python test/calibrate_scaling.py --robot shadow --input mediapipe --video data/right.mp4

# Calibrate with Vision Pro
python test/calibrate_scaling.py --robot wuji --input avp --avp-ip 192.168.5.32

# Calibrate with Noitom
python test/calibrate_scaling.py --robot inspire --input noitom

# Calibrate with Quest 3
python test/calibrate_scaling.py --robot shadow --input quest3

visualize_scaling.py

Visualize how scaling and segment_scaling parameters affect MediaPipe keypoints.

cd example

python test/visualize_scaling.py --robot leap --video data/right.mp4 --hand right
python test/visualize_scaling.py --robot allegro --play data/avp1.pkl --hand right

API Reference

Basic Usage

from anydexretarget import Retargeter

# Load from config file
retargeter = Retargeter.from_yaml("config/adaptive/mediapipe/mediapipe_shadow_hand.yaml", hand_side="right")

# Retarget: (21, 3) MediaPipe keypoints -> joint angles
qpos = retargeter.retarget(raw_keypoints)

# With verbose output
qpos, info = retargeter.retarget_verbose(raw_keypoints)
print(f"Cost: {info['cost']:.4f}")
print(f"Pinch alphas: {info.get('pinch_alphas')}")  # adaptive only

Advanced Usage

# Direct optimizer access
optimizer = retargeter.optimizer

# Compute cost for given pose
cost = optimizer.compute_cost(qpos, mediapipe_keypoints)

# Get timing statistics
stats = optimizer.get_timing_stats()
print(f"Average time: {stats.get_avg()['total_ms']:.2f} ms")

Citation

@software{anydexretarget2025,
  title={AnyDexRetarget},
  author={Shiquan Qiu},
  year={2025},
  url={https://gitee.com/gx_robot/AnyDexRetarget},
}

Acknowledgement

MuJoCo - Physics simulation
MuJoCo Menagerie - Shadow Hand models
dex-retargeting - Retargeting algorithms
DexPilot - Vision-based teleoperation
VisionProTeleop - Apple Vision Pro streaming
wuji-retargeting - Wuji retargeting

Contact

For questions, please open an issue on Gitee / GitHub or contact the author via 932851972@qq.com.

Name		Name	Last commit message	Last commit date
Latest commit History 28 Commits
anydexretarget		anydexretarget
assets		assets
example		example
.gitattributes		.gitattributes
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
README.zh.md		README.zh.md
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
setup.py		setup.py

Folders and files

Latest commit

History

Repository files navigation

AnyDexRetarget

Demo

Simulation Retargeting

Quest 3 Hand-Arm Teleoperation

Features

Table of Contents

Supported Robots

Repository Structure

Installation

Prerequisites

Install

Troubleshooting

Quick Start

Simulation

Real Hardware

Command Reference

Input Source

Optimizer

Robot Hand & Output

Debug & Visualization Tools

debug_skeleton.py

calibrate_scaling.py

visualize_scaling.py

API Reference

Basic Usage

Advanced Usage

Citation

Acknowledgement

Contact

About

Topics

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases 1

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages