ClothCap

High-Fidelity 4D Cloth Capture Pipeline with a Two-Level Pattern.

Further details and data are available on the project page.

Supported garments: suit, shirt, skirt, pants
Camera setup: up to 16 synchronized cameras (A–P)

Pipeline overview

Multi-camera images
      ↓
1. Marker detection       (ResNet18, dilated conv)
      ↓
2. Marker recognition     (ResNet50 per garment)
      ↓
3. UV localization        (ResNet50 atrous)
      ↓
4. 2D mesh recovery       (Warp GPU kernels)
      ↓  [2d_localize.py]
5. Multi-view triangulation  (C++/Ceres, CUDA sparse)
      ↓  [triangulate.py]
6. 3D reconstruction      (Newton physics simulation)
      ↓  [inpaint.py]
3D garment mesh (.npy)

Installation

1. Python environment

# core dependencies (equivalently: pip install -e . once the extension is built)
pip install torch torchvision kornia h5py numpy Pillow tqdm pyyaml opencv-python libigl \
    "warp-lang>=1.14" "newton>=1.3"

Optional extras:

pip install polyscope     # live viewer for `inpaint.py --vis`

2. System dependencies for the triangulation extension

The triangulation stage is a C++ pybind11 extension using Ceres Solver and TBB. The following must be installed system-wide before building:

TBB and OpenCV (Ubuntu/Debian):

sudo apt install libtbb-dev libopencv-dev

Ceres Solver — build from source (the packaged version is usually too old):

# SuiteSparse and other Ceres dependencies
sudo apt install libgoogle-glog-dev libsuitesparse-dev libatlas-base-dev

git clone https://github.com/ceres-solver/ceres-solver
cd ceres-solver && mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DBUILD_EXAMPLES=OFF \
         -DCUDA=ON   # omit if no GPU
make -j$(nproc) && sudo make install

Ceres is installed to /usr/local by default, which is where CMakeLists.txt looks for it.

CUDA toolkit — required for the GPU solver (skip for CPU-only):

# Follow https://developer.nvidia.com/cuda-downloads for your distro.
# CUDA 11+ is sufficient; the extension links against libcusparse/libcublas/libcusolver.

Conan 2.x — manages spdlog and Eigen:

pip install conan

3. Build the triangulation extension

Step 1 — generate Conan dependencies:

conan install cpp --output-folder=cpp/build --build=missing -pr cpp/profiles/linux-release

Step 2 — build and install:

pip install -e . --no-build-isolation

This compiles _triangulate.so and installs it into your Python environment. triangulate.py imports it automatically; if the build is skipped it falls back to a pre-built binary.

GPU (CUDA) solver — optional. By default the build uses the CPU solver (USE_CUDA_SPARSE=OFF). The Conan ceres-solver package ships without CUDA support on every platform, so the GPU CUDA_SPARSE path is unavailable through Conan — you must supply a Ceres built with CUDA yourself and point the build at it via the ceres_from=external / ceres_prefix Conan options.

GPU build (Windows)

Requires the CUDA Toolkit 12.x (provides cuSPARSE/cuSOLVER/cuBLAS) and Visual Studio 2022. The steps below assume a vcpkg at D:\code\vcpkg.

1. Build Ceres with CUDA via vcpkg (also pulls Eigen/glog/gflags/SuiteSparse):

   D:\code\vcpkg\vcpkg.exe install "ceres[cuda,suitesparse]:x64-windows"

2. Generate Conan deps pointing at the external Ceres (its install prefix, not the share/ceres dir):

   conan install cpp --output-folder=cpp/build --build=missing -pr .\cpp\profiles\windows-release `
     -o "&:ceres_from=external" `
     -o "&:ceres_prefix=D:/code/vcpkg/installed/x64-windows"

3. Build the extension with the CUDA solver enabled:

   pip install -e . --no-build-isolation "--config-settings=cmake.args=-DUSE_CUDA_SPARSE=ON"

The build automatically (a) wires up vcpkg's glog/gflags target naming, (b) copies the vcpkg runtime DLLs next to the extension, and (c) triangulate.py adds %CUDA_PATH%\bin to the DLL search path at import (Python 3.8+ no longer searches PATH for an extension's dependent DLLs), so the CUDA runtime resolves without copying it. CUDA_PATH is set by the CUDA Toolkit installer.

Note: the native module is loaded through triangulate.py, which installs the CUDA DLL-search hook. A bare import _triangulate won't have it and will fail with DLL load failed — replicate the os.add_dll_directory call if you import the native module elsewhere.

GPU build (Linux)

Build Ceres from source with -DUSE_CUDA=ON (e.g. installed to /usr/local, as in step 2 above), then point the build at it the same way:

conan install cpp --output-folder=cpp/build --build=missing -pr cpp/profiles/linux-release \
  -o "&:ceres_from=external" -o "&:ceres_prefix=/usr/local"
pip install -e . --no-build-isolation -Ccmake.args="-DUSE_CUDA_SPARSE=ON"

4. Newton physics engine

Stage 6 uses NVIDIA Newton (installed as part of the core dependencies above). To install it on its own:

pip install "newton>=1.3"

Data and model weights

Download from Google Drive and place under data/ and models/:

• Model weights (detection, per-garment recognition, UV localization): [Google Drive link]
• Garment data files (*.hdf5, *.npz, *.npy, *.obj): [Google Drive link]
• Camera calibration (camera_params_*.json): [Google Drive link]

Expected layout:

data/
├── camera_params_pants_C.json
├── camera_params_suit_sim.json
├── ...
├── suit/
│   ├── suit_crop_info.hdf5
│   ├── suit_mesh_crop_info.hdf5
│   ├── suit_pixel.npz
│   ├── suit_marker_info.hdf5
│   ├── suit_mesh_neighbor.npy
│   └── suit_coarse6.obj
├── shirt/  ...
├── skirt/  ...
└── pants/  ...

models/
├── marker_detection.pth
├── suit_recognition.pth
├── shirt_recognition.pth
├── skirt_recognition.pth
├── pants_recognition.pth
└── uv_localization.pth

Usage

The pipeline is three scripts run in order. Each takes --config config.yaml and a --sequence naming one of the blocks under sequences: in the config; a sequence's fields override the shared pipeline defaults. Outputs are written under that sequence's output_root, so any stage can resume from a previous run's outputs.

Using pants_C (garments shirt, pants) as the running example:

Stage 1–4 — 2D localization (2d_localize.py)

Runs marker detection, recognition, UV localization, and 2D mesh recovery, writing per-stage dumps under {output_root}/stage*/ and the triangulation inputs under {output_root}/coords_2d/.

python 2d_localize.py --config config.yaml --sequence pants_C

Flag	Description
--config PATH	YAML config (required).
--sequence NAME	Sequence block to load; omit to use bare pipeline defaults.
--cuda N	Override the config GPU index (-1 for CPU).
--vis-det / --vis-recog / --vis-uv	Write detection / recognition / UV+2D-mesh visualizations to {output_root}/stage*/vis/.

# with all visualizations
python 2d_localize.py --config config.yaml --sequence pants_C \
    --vis-det --vis-recog --vis-uv

Stage 5 — Multi-view triangulation (triangulate.py)

Consumes {output_root}/coords_2d/ and writes triangulated 3D marker positions to {output_root}/coords_3d/{garment}/ (one .npz per frame).

# all garments in the sequence
python triangulate.py --config config.yaml --sequence pants_C

# a single garment
python triangulate.py --config config.yaml --sequence pants_C --garment pants

Flag	Description
--config PATH	YAML config (required).
--sequence NAME	Sequence block to load.
--garment NAME	Triangulate only this garment (default: every garment in the sequence).

Stage 6 — 3D reconstruction (inpaint.py)

Physics-based multigrid reconstruction. Each invocation runs one multigrid level (--level is required). Levels are reconstructed coarse-to-fine and each level initializes from the previous one, so run them in ascending order:

python inpaint.py --config config.yaml --sequence pants_C --level 1
python inpaint.py --config config.yaml --sequence pants_C --level 3
python inpaint.py --config config.yaml --sequence pants_C --level 6
python inpaint.py --config config.yaml --sequence pants_C --level 12

Level 1 seeds from the sequence's init_file; levels 3, 6, 12 upscale from the previous level via the init_interp_file mapping. The available levels per garment are defined in the recover_3d.{sequence}.{garment}.levels block of the config (not every garment defines all four — e.g. suit stops at 6).

Flag	Description
--config PATH	YAML config (required).
--sequence NAME	Sequence block to load (required).
--level {1,3,6,12}	Single multigrid level to run (required; --levels accepted as an alias).
--garment NAME	Reconstruct only this garment (default: every garment in the sequence).
--vis	Show a live Polyscope viewer during solve (requires a display and the optional polyscope dependency).
--cuda N	Override the config GPU index (-1 for CPU).

End-to-end example

python 2d_localize.py --config config.yaml --sequence pants_C
python triangulate.py --config config.yaml --sequence pants_C
for L in 1 3 6 12; do
    python inpaint.py --config config.yaml --sequence pants_C --level $L
done

Config reference

See config.yaml for all options with inline comments.

Sequences

Add one entry per capture session under sequences:. At minimum set image_dir, output_root, garments, start_frame, end_frame, and camera_params_file. Any field in a sequence block overrides the matching pipeline default; the --sequence flag selects which block to merge in.

For Stage 6, add a matching entry under recover_3d.{sequence} describing each garment's multigrid levels (rest shapes, initialization/interpolation files, and solver parameters).

Key fields

Field	Scope	Description
image_dir	sequence	Root directory of camera image sequences
output_root	sequence	Root output directory; all stage outputs derive from it
garments	sequence	List of garments to process (e.g. [shirt, pants])
start_frame / end_frame	sequence	Frame range (end exclusive)
camera_params_file	sequence	Path to camera calibration JSON
recog_threshold	sequence/pipeline	Marker recognition acceptance threshold
cameras	pipeline	Camera ID string (default: ABCDEFGHIJKLMNOP, up to 16)
cuda	pipeline	GPU device index (-1 for CPU)
detection_model_path	pipeline	Marker detection weights
recover_3d.{seq}.{garment}.levels	recover_3d	Per-level multigrid solver config for Stage 6

Output layout

All outputs are derived from pipeline.output_root:

{output_root}/
├── stage1_detection/        marker detection dumps + vis/
├── stage2_recognition/      recognition dumps + vis/
├── stage3_uv/               UV localization dumps + vis/
├── stage4_mesh2d/           2D mesh recovery vis/
├── coords_2d/{garment}/     triangulation inputs  (per frame: {cam}{frame:05d}.npz)
├── coords_3d/{garment}/     triangulation outputs (per frame: {frame:05d}.npz)
└── dense/{garment}/         3D reconstruction outputs

Citation

If you use this code, please cite:

@article{10.1145/3811305,
author = {Liu, Ziheng and Chen, Anka and Chen, Shu and Yang, Yin and Yuksel, Cem and Lin, Jenny Han},
title = {High-Fidelity 4D Cloth Capture Pipeline with a Two-Level Pattern},
year = {2026},
issue_date = {July 2026},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {45},
number = {4},
issn = {0730-0301},
url = {https://doi.org/10.1145/3811305},
doi = {10.1145/3811305},
journal = {ACM Trans. Graph.},
month = jul,
articleno = {111},
numpages = {12}
}

License

MIT — see LICENSE.