Skip to content

Add fvdb.nn.Prune and generative shape completion / VAE examples - #753

Open
swahtz wants to merge 7 commits into
openvdb:mainfrom
swahtz:feature/prune-layer-generative-examples
Open

Add fvdb.nn.Prune and generative shape completion / VAE examples#753
swahtz wants to merge 7 commits into
openvdb:mainfrom
swahtz:feature/prune-layer-generative-examples

Conversation

@swahtz

@swahtz swahtz commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Addresses #741.

Summary

  • fvdb.nn.Prune — the MinkowskiPruning analog: prunes grid topology and its aligned features by a per-voxel boolean mask (pruned_grid + rmask, differentiable through kept rows). Registered in docs; 13 unit tests.
  • examples/shape_completion.py / examples/shape_vae.py — one-to-one analogs of MinkowskiEngine's completion.py and vae.py, built on the generative transposed convolution from Unify sparse convolution and transpose semantics #726 (from_grid_batch_transposed(..., target_grid=None)) plus Prune.
  • notebooks/04_shape_vae.ipynb — trains the VAE and explores its latent space: reconstructions, prior sampling, latent interpolation (ballet flat → boot), and an optional ipywidgets explorer. Shapes render as lit voxel surfaces.
  • Dataset: the VAE trains on the 254-mesh GSO "Shoe" subset (CC-BY 4.0, Add Google Scanned Objects 'Shoe' subset (254 meshes, CC-BY 4.0) voxel-foundation/fvdb-example-data#1) via a new load_gso_shoes() loader and data-revision bump; completion uses the dragon + happy meshes.
  • Removes examples/wip/structure_prediction_net.py (targeted a removed API surface; superseded by the new examples).

Design notes

  • Per-level GT targets come from a conv_grid(2, 2) pyramid queried with coords_in_grid — not coarsened_grid, whose block-centroid lattice differs from what conv_transpose_grid(2, 2) inverts.
  • ME's teacher forcing (keep |= target) and per-level BCE are reproduced verbatim; eval decodes without forcing.
  • The VAE's dense-neck seed adds a learned per-voxel positional embedding — a uniform broadcast latent leaves neck voxels spatially indistinguishable after k2 transposes.
  • Generative training is bound by per-iteration generated-topology grid construction (kernel maps are cheap); tracked upstream as Generated-topology grid construction dominates per-iteration ConvolutionPlan cost (serial per-grid build + merge) #755 rather than worked around here.

Results / verification

  • Completion IoU ≈ 0.75 (500 iters, ~40 s); VAE reconstruction IoU ≈ 0.49 on a 254-shape dataset (1500 iters, ~3.5 min), prior samples decode to plausible shoes.
  • pytest tests/unit/test_nn_modules.py: 81 passed (CPU+CUDA, float32/64).
  • pytest tests/test_examples.py -k shape: 2 passed (~4 min, CI runpy path); notebook passes pytest --nbmake.
  • black --line-length 120 clean.

🤖 Generated with Claude Code

Add a Prune module to fvdb.nn (the MinkowskiPruning analog): prunes grid
topology and its aligned features by a per-voxel boolean mask, built on
GridBatch.pruned_grid + JaggedTensor.rmask, differentiable through the
kept feature rows.

Add two examples mapping one-to-one onto MinkowskiEngine's canonical
generative examples, built on the generative transposed convolution
semantics from openvdb#726 (ConvolutionPlan.from_grid_batch_transposed with
target_grid=None):

- examples/shape_completion.py (analog of ME examples/completion.py):
  sparse encoder-decoder completing a slab-cropped shape, per-level
  occupancy classifiers, teacher-forced pruning, additive U-Net skips
  via inject_from. GT level targets come from a conv_grid(2, 2) pyramid
  queried with coords_in_grid.
- examples/shape_vae.py (analog of ME examples/vae.py): shape VAE with
  a jagged global-pooling latent bottleneck, dense-neck decoder seeded
  from the latent plus a learned positional embedding, KL + per-level
  BCE loss, and prior-sampling demo.

Both examples train on bundled fvdb-example-data meshes in ~1-2 minutes
and run headlessly under tests/test_examples.py.

Addresses openvdb#741.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz
swahtz requested a review from a team as a code owner August 31, 2026 23:19
@swahtz
swahtz requested review from phapalova and sifakis August 31, 2026 23:19
notebooks/04_shape_vae.ipynb trains the shape VAE from
examples/shape_vae.py and explores its latent space: reconstructions,
prior sampling, batched latent interpolation, and an ipywidgets-based
interactive latent explorer (blend between shapes, add prior noise)
with a static fallback when ipywidgets is unavailable (e.g. in CI).

Executed outputs are stored so the notebook renders on GitHub
(reconstruction IoU 0.999; tqdm progress spam stripped). Runs under
pytest --nbmake in ~87 s on GPU. Adds ipywidgets to the fvdb_learn
environment for the interactive path.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz

swahtz commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Added notebooks/04_shape_vae.ipynb: trains the shape VAE from examples/shape_vae.py and explores its latent space interactively — reconstructions (IoU 0.999), prior sampling, batched latent interpolation (car → dragon), and an ipywidgets latent explorer (blend two shapes + prior noise) with a static fallback when ipywidgets is unavailable. Executed outputs are stored so it renders on GitHub; passes pytest --nbmake in ~87 s on GPU. ipywidgets added to the fvdb_learn environment.

Switch examples/shape_vae.py and the sampling notebook from the 6
bundled meshes to the 254-model Google Scanned Objects 'Shoe' subset
(CC-BY 4.0) now in fvdb-example-data: bump the pinned data revision and
add a load_gso_shoes() loader to fvdb.utils.examples.

One object category with real intra-class variation (runners, flats,
cleats, boots) makes latent interpolation and prior sampling meaningful,
mirroring the role of ModelNet40 chairs in MinkowskiEngine's vae.py.
Training now runs random minibatches over the pre-voxelized dataset
(GridBatch indexing); the ground-truth conv_grid pyramid is built once
and sub-indexed per batch. The notebook interpolates a ballet flat into
a tall boot and uses a side-on camera.

Generative training remains bound by per-iteration generated-topology
grid construction (conv_transpose_grid); tracked upstream as openvdb#755.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz

swahtz commented Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Dataset upgrade: the shape VAE (example + notebook) now trains on the 254-model Google Scanned Objects Shoe subset (CC-BY 4.0, added to fvdb-example-data in voxel-foundation/fvdb-example-data#1) instead of the 6 bundled meshes — one category with real intra-class variation, mirroring the role of ModelNet40 chairs in ME's vae.py. Adds fvdb.utils.examples.load_gso_shoes() and bumps the pinned data revision; training runs random minibatches over the pre-voxelized GridBatch with a once-built conv_grid pyramid sub-indexed per batch. The notebook now interpolates a ballet flat into a tall boot.

CI note: pytest tests/test_examples.py -k shape now takes ~4:07 (was ~2:02) due to the larger dataset and minibatch training.

Perf note: generative training is bound by per-iteration generated-topology grid construction (conv_transpose_grid; the kernel maps themselves are cheap) — profiled and filed as #755 rather than worked around here, so the examples stay semantically 1:1 with MinkowskiEngine.

Replace the matplotlib point scatters in the shape-VAE notebook with lit
voxel-cube renders: exposed cube faces from pcu.voxel_grid_geometry
(interior faces between adjacent voxels culled via duplicate-centroid
removal), per-face Lambertian shading, per-panel auto-fit equal-aspect
bounds, and unswapped z-up axes (the GSO scans follow the Gazebo z-up
convention; the previous y-up swap was showing the shapes sole-on).

Shapes are now clearly readable as shoes - laces, soles, and boot
shafts are visible in the dataset, reconstruction, sampling, and
interpolation figures.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
@swahtz swahtz added this to the v0.6 milestone Sep 2, 2026
@swahtz swahtz added documentation Improvements or additions to documentation core library Core fVDB library. i.e. anything in the _Cpp module (C++) or fvdb python module Topology Operations Issues related to topology operations (prune, merge, dilate, etc. labels Sep 2, 2026
@swahtz
swahtz marked this pull request as ready for review September 2, 2026 04:15
examples/wip/structure_prediction_net.py targeted a removed API surface
(VDBTensor, fvnn.ReLU, fvnn.FillFromGrid, per-module conv backends) and
undeclared dependencies (torch_cudamanaged, hardcoded local data paths),
so it no longer runs. Its intent - hierarchical structure prediction
with per-level occupancy classifiers, losses against a coarsened
ground-truth pyramid, U-Net skips onto predicted topology, and a dense
neck - is now covered by the maintained, CI-tested examples
shape_completion.py and shape_vae.py, which use the generative
transposed-convolution path from openvdb#726 plus fvdb.nn.Prune.

Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>

@Rishit-dagli Rishit-dagli left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are there any plans for a similar shape vae example but for 4d volume?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core library Core fVDB library. i.e. anything in the _Cpp module (C++) or fvdb python module documentation Improvements or additions to documentation Topology Operations Issues related to topology operations (prune, merge, dilate, etc.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants