DynaHOI: Benchmarking Hand-Object Interaction for Dynamic Target

BoCheng Hu ¹,  Zhonghan Zhao ^1,2,  Kaiyue Zhou ³,  Hongwei Wang ¹  Gaoang Wang ¹ 

¹ Zhejiang University  ² Shanghai AI Lab  ³ Chengdu Minto Technology Co., Ltd. 

News 📣

• ✅ We’ve released the current evaluation / inference code used in our experiments.
• Feb 13, 2026: 🔥 We open-sourced ObAct weights.
• Feb 11, 2026: 📄 Our paper is now public.

TODO ✅

• Paper release
• Model weights release (ObAct)
• Dataset release
• Training code release
• Release dynamic HOI trajectory collection scripts & tutorial

Setup ⚙️

1) Unity Environment (Simulator) 🎮

This project uses a Unity simulator for two purposes:

• Dynamic HOI trajectory collection
• Online (closed-loop) evaluation with a policy running in Python

Steps

1. Install Unity Hub, then install Unity 2022.3.58f1c1.
2. Download the full Unity project (C# source + required assets) from Google Drive, Unity Controller folder.
3. Open the project in Unity Hub, then double-click the Scenes/Dynamic scene file to load the predefined scene in the Unity Editor. That’s it — your simulator should be ready ✅

---

2) Python Environment (Conda) 🐍

conda create -n gr00t python=3.10
conda activate gr00t
pip install --upgrade setuptools
pip install -e .[base]
pip install --no-build-isolation flash-attn==2.7.1.post4

We also recommend using uv for faster & cleaner dependency management. You can follow NVIDIA’s official setup guide here.

Online Evaluation 🧪 (Closed-loop)

Overview 🔁

• Python runs the policy inference and acts as the server (actively connects).
• Unity executes action logic and acts as the client (waits for Python connection).

Since inference and simulation often run on different machines (e.g., Unity on your local desktop, inference on a remote GPU server), you’ll likely need port forwarding.

Steps

1. SSH port forwarding (remote ↔ local):

ssh -L 8765:127.0.0.1:8765 <your_remote_server>

2. Download ObAct weights from Google Drive.Then update model_path in the inference script:

   • scripts/eval_policy_baseline.py → ObAct.

   • scripts/eval_policy_our.py → vanilla GR00T-N1.5.

3. Run the policy-side inference:

python scripts/eval_policy_baseline.py
# or
python scripts/eval_policy_our.py

4. In Unity, click Play ▶️ in the GUI.

You should see real-time interaction in the Game window.

Fine-tuning 🛠️

Script mapping:

• Vanilla GR00T-N1.5 fine-tuning:

  scripts/gr00t_finetune_our_18dim.py

• ObAct baseline fine-tuning:

  scripts/gr00t_finetune_our_18dim_baseline.py

Before running, configure key args like:

• dataset_path
• output_dir
• base_model_path
• (and any cluster / logging configs you use)

We recommend launching with setsid + torchrun:

setsid torchrun --standalone --nproc_per_node=4 scripts/gr00t_finetune_our_18dim.py > nohup_logs/xxx.log 2>&1 &

Dynamic HOI Trajectory Collection 📹🤖

We provide:

• atomic motion generation scripts
• automated grasping trajectory collection scripts

in the Google Drive, Unity Controller folder:

A detailed tutorial is coming soon, including:

• how to generate large-scale, valid motion parameters
• how to collect data via WebSocket while compressing videos
• recommended Unity simulator settings (scene setup, playback rate, resolution, logging, etc.)

Citation 📚

@article{hu2026dynahoi,
  title={DynaHOI: Benchmarking Hand-Object Interaction for Dynamic Target},
  author={Bocheng Hu and Zhonghan zhao and Kaiyue zhou and Hongwei Wang and Gaoang Wang},
  journal={arXiv preprint arXiv:2602.11919},
  year={2026}
}

Acknowledgements 🙌✨

Huge thanks to these excellent open-source projects that made this work possible:

GR00T, OpenVLA, UP-VLA, Openpi, Lerobot and OmniObject3D.