3DreamBooth

High-Fidelity 3D Subject-Driven Video Generation

Hyun-kyu Ko^1,*  ·  Jihyeon Park^2,*  ·  Younghyun Kim¹  ·  Dongheok Park²  ·  Eunbyung Park<sup>1,†

1</sup> Yonsei University    ² Sungkyunkwan University
_{* Equal contribution    † Corresponding author}

teaser.mp4

---

Overview

3DreamBooth is a novel framework for high-fidelity 3D subject-driven video generation. Given a set of multi-view reference images of a subject, our method generates identity-preserving, view-consistent videos with rich 3D spatial awareness.

Our framework comprises two components:

• 3Dapter — a visual conditioning module that enhances fine-grained texture preservation and accelerates convergence via multi-view joint attention with shared weights.
• 3DreamBooth — a DreamBooth-style test-time optimization that decouples spatial geometry from temporal motion through a 1-frame optimization paradigm, baking a robust 3D prior without exhaustive video-based training.

---

Results

3DreamBooth generates cinematic, identity-preserving videos across diverse subjects and creative scenarios — bags, plushies, sculptures, motorcycles, watches, and more.

📽️ See our project page for full video results.

---

Code Release

The public release currently includes:

• ✅ Inference code
• ✅ 3Dapter pretrained weights
• ✅ 3D-CustomBench benchmark
• ✅ Training code
• ⬜ Evaluation code

Star ⭐ this repo to get notified when the evaluation code drops.

---

Training and inference

Mode	Train	Validate	What it learns
3DreamBooth	train_3dreambooth.py	validate_3dreambooth.py	Subject-specific spatial LoRA
3Dapter	train_3dapter.py	validate_3dapter.py	Reference-conditioned adapter
Joint	train_joint.py	validate_joint.py	Subject LoRA + 3Dapter together

All three modes use the same YAML-driven interface:

python scripts/run.py <config.yaml>

The runner resolves config inheritance, applies command-line overrides, launches torch.distributed.run, and prints the exact command for reproducibility.

Installation

git clone https://github.com/Ko-Lani/3DreamBooth.git
cd 3DreamBooth

conda create -n 3dreambooth python=3.10 -y
conda activate 3dreambooth
pip install -r requirements.txt

Python 3.10 and a recent CUDA-capable PyTorch environment are recommended. Model and adapter weights are intentionally not included in this repository.

Checkpoint setup

Install the Hugging Face and ModelScope CLIs:

pip install -U "huggingface_hub[cli]" modelscope

Download HunyuanVideo-1.5 into the path used by the public configs:

hf download tencent/HunyuanVideo-1.5 \
  --local-dir ./checkpoints/hunyuanvideo-1.5 \
  --include "transformer/480p_t2v*" "transformer/720p_t2v*" \
  "text_encoder/**" "tokenizer/**" "vae/**" "*.json" "*.txt" "*.md"

hf download Qwen/Qwen2.5-VL-7B-Instruct \
  --local-dir ./checkpoints/hunyuanvideo-1.5/text_encoder/llm
hf download google/byt5-small \
  --local-dir ./checkpoints/hunyuanvideo-1.5/text_encoder/byt5-small
modelscope download --model AI-ModelScope/Glyph-SDXL-v2 \
  --local_dir ./checkpoints/hunyuanvideo-1.5/text_encoder/Glyph-SDXL-v2

The vision encoder requires access to FLUX.1-Redux-dev:

hf download black-forest-labs/FLUX.1-Redux-dev \
  --local-dir ./checkpoints/hunyuanvideo-1.5/vision_encoder/siglip \
  --token YOUR_HF_TOKEN

Download the pretrained 3Dapter used by Joint training:

hf download lanikoisgod/3Dapter --local-dir ./checkpoints/3dapter

Expected local layout:

checkpoints/
├── hunyuanvideo-1.5/
│   ├── transformer/
│   ├── text_encoder/
│   ├── tokenizer/
│   ├── vae/
│   └── vision_encoder/
└── 3dapter/
    └── pytorch_lora_weights.safetensors

Pretrained CustomBench checkpoints

We also release ready-to-use joint (3DreamBooth + 3Dapter) LoRA checkpoints for all 30 subjects in 3D-CustomBench, so you can run inference without training from scratch: lanikoworld/3DreamBooth-CustomBench.

hf download lanikoworld/3DreamBooth-CustomBench \
  --local-dir ./checkpoints/custombench --include "graduation_bear/*"

python validate_joint.py \
  --pretrained_model_root ./checkpoints/hunyuanvideo-1.5 \
  --pretrained_transformer_version 720p_t2v \
  --subject_adapter_path ./checkpoints/custombench/graduation_bear/subject_adapter/pytorch_lora_weights.safetensors \
  --tdapter_path ./checkpoints/custombench/graduation_bear/tdapter/pytorch_lora_weights.safetensors \
  --reference_path ./datasets/3d-custombench/subjects/graduation_bear/references \
  --prompt "A video of a rhs bear on a beach." \
  --text_lora_spans "rhs bear" \
  --video_length 81

Swap graduation_bear for any 3D-CustomBench subject id to use that checkpoint instead.

Each checkpoint has its own trigger phrase: the identifier rhs plus a class word that differs per subject (rhs bear, rhs mug, rhs bust, ...). The phrase must appear verbatim in the prompt and be marked as the LoRA span, or the subject identity will not activate. Look it up in subjects.json:

import json
from huggingface_hub import hf_hub_download

meta = json.load(open(hf_hub_download(
    "lanikoworld/3DreamBooth-CustomBench", "subjects.json")))
spans = {s["subject_id"]: s["lora_span"] for s in meta["subjects"]}
print(spans["graduation_bear"])   # "rhs bear"

Each checkpoint also ships the training_config.json it was trained with.

Quick start: 3D-CustomBench graduation bear

1. Download 3D-CustomBench

python scripts/data/download_custombench.py

This downloads lanikoworld/3D-CustomBench to datasets/3d-custombench. The example expects:

datasets/3d-custombench/subjects/graduation_bear/
├── images/
├── references/
├── metadata.json
└── prompt.txt

Use a different dataset fork or revision when needed:

python scripts/data/download_custombench.py \
  --repo-id YOUR_ORG/3D-CustomBench \
  --revision main

2. Inspect the resolved commands

Dry-run mode loads and validates the YAML interface without loading model weights:

python scripts/run.py configs/examples/graduation_bear/train_3dreambooth.yaml --dry-run
python scripts/run.py configs/examples/graduation_bear/train_joint.yaml --dry-run

3. Train

Train the 3DreamBooth baseline:

python scripts/run.py configs/examples/graduation_bear/train_3dreambooth.yaml

Train the full Joint model:

python scripts/run.py configs/examples/graduation_bear/train_joint.yaml

The canonical prompt is A video of a [v] [class].; the graduation-bear example uses A video of a [rhs bear].. Write the LoRA phrase inside square brackets—there is no need to configure span arguments separately.

How do bracketed LoRA spans work?

Square brackets are config-only markup. They are always removed before the prompt reaches the tokenizer.

1. Training

train_prompts:
  - A video of a [rhs bear].

The runner sends the clean prompt below and applies LoRA to the full training prompt:

A video of a rhs bear.

2. Validation during training

validation_prompts:
  - A video of a [rhs bear] on a beach.

The runner removes the brackets and automatically creates the legacy argument:

--validation_prompts "A video of a rhs bear on a beach." \
--validation_lora_spans "rhs bear"

3. Validation/inference after training

prompt: A video of a [rhs bear] on a beach.

The runner converts it to:

--prompt "A video of a rhs bear on a beach." \
--text_lora_spans "rhs bear"

The model receives the clean sentence, but text LoRA is applied only to the rhs bear tokens. The surrounding scene description does not receive text LoRA. Unbalanced brackets and validation prompts without a marked span are rejected before model loading.

See Prompt and LoRA span format for multiple-span examples.

4. Validate

After training reaches checkpoint-400:

python scripts/run.py configs/examples/graduation_bear/validate_3dreambooth.yaml
python scripts/run.py configs/examples/graduation_bear/validate_joint.yaml

Validate the pretrained 3Dapter independently:

python scripts/run.py configs/examples/graduation_bear/validate_3dapter.yaml

Generated videos are written below validation_outputs/ and ignored by Git.

Configuration

Base configs live in:

configs/
├── train/
│   ├── 3dreambooth.yaml
│   ├── 3dapter.yaml
│   └── joint.yaml
├── validate/
│   ├── 3dreambooth.yaml
│   ├── 3dapter.yaml
│   └── joint.yaml
└── examples/graduation_bear/

Each file has the same shape:

name: my_experiment
stage: train                 # train | validate
method: joint                # 3dreambooth | 3dapter | joint

environment:
  CUDA_VISIBLE_DEVICES: "0"

runtime:
  nproc_per_node: 1
  master_port: 8003

args:                        # forwarded to the selected Python entrypoint
  train_prompts:
    - A video of a [rhs plushie].
  max_steps: 400
  batch_size: 1

Override any value without editing the file:

python scripts/run.py configs/train/joint.yaml \
  --set args.instance_data_root=./data/my_subject/images \
  --set args.reference_path=./data/my_subject/references \
  --set args.output_dir=./outputs/joint/my_subject \
  --set args.max_steps=800

For multi-GPU sequence parallelism, set both process count and sequence-parallel size:

python scripts/run.py configs/train/joint.yaml \
  --set runtime.nproc_per_node=4 \
  --set args.sp_size=4

See Training and validation modes for method-specific inputs.

Prepare your own subject

Use the same layout as a 3D-CustomBench subject:

data/my_subject/
├── images/
│   ├── 001.jpeg
│   └── ...
└── references/
    ├── 001.png
    └── ...

images/ contains the ordered multi-view captures. references/ contains a small subset with the background removed, composited on white, and normalized to a square crop for 3Dapter conditioning. The current preprocessing implementation is in remove_bg.py; set its input/output paths for your subject before running it.

DreamBooth prompts use a rare identifier followed by a class word inside square brackets:

A video of a [rhs plushie].

Square brackets are config-only markup:

• Training removes the brackets and applies LoRA to the full prompt.
• Validation during training generates validation_lora_spans from the marked phrase.
• Validation/inference after training generates text_lora_spans from the marked phrase.

The class word (plushie) describes the category; the rare identifier (rhs) carries its learned identity. Keep the marked phrase consistent across training and validation.

Repository layout

3DreamBooth/
├── configs/                  # reproducible train/validation YAML files
├── docs/                     # detailed usage and release notes
├── hyvideo/                  # models, pipelines, and datasets
├── scripts/
│   ├── data/                 # 3D-CustomBench download utilities
│   ├── train/                # thin config-based shell entrypoints
│   ├── validate/             # thin config-based shell entrypoints
│   └── run.py                # unified experiment launcher
├── train_*.py                # core training implementations
└── validate_*.py             # core validation implementations

Notes

• Video lengths used by 3DreamBooth and Joint validation must be 4n + 1 (for example 49, 81, or 129).

License

Original 3DreamBooth contributions are released under the Apache License 2.0. This repository contains code derived from HunyuanVideo-1.5, which remains subject to the Tencent Hunyuan Community License. See NOTICE for third-party attributions.

3D-CustomBench is released under CC BY 4.0.

Citation

If this project is useful in your research, please cite the paper:

@misc{ko20263dreambooth,
  title         = {3DreamBooth: High-Fidelity 3D Subject-Driven Video Generation Model},
  author        = {Hyun-kyu Ko and Jihyeon Park and Younghyun Kim and Dongheok Park and Eunbyung Park},
  year          = {2026},
  eprint        = {2603.18524},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2603.18524}
}

Acknowledgements

This project builds upon HunyuanVideo-1.5. Our subject-customization formulation is inspired by DreamBooth, and our reference-conditioning design builds on OminiControl.

We thank the authors for their excellent work and open-source contributions.