Add fvdb.nn.Prune and generative shape completion / VAE examples - #753
Add fvdb.nn.Prune and generative shape completion / VAE examples#753swahtz wants to merge 7 commits into
Conversation
Add a Prune module to fvdb.nn (the MinkowskiPruning analog): prunes grid topology and its aligned features by a per-voxel boolean mask, built on GridBatch.pruned_grid + JaggedTensor.rmask, differentiable through the kept feature rows. Add two examples mapping one-to-one onto MinkowskiEngine's canonical generative examples, built on the generative transposed convolution semantics from openvdb#726 (ConvolutionPlan.from_grid_batch_transposed with target_grid=None): - examples/shape_completion.py (analog of ME examples/completion.py): sparse encoder-decoder completing a slab-cropped shape, per-level occupancy classifiers, teacher-forced pruning, additive U-Net skips via inject_from. GT level targets come from a conv_grid(2, 2) pyramid queried with coords_in_grid. - examples/shape_vae.py (analog of ME examples/vae.py): shape VAE with a jagged global-pooling latent bottleneck, dense-neck decoder seeded from the latent plus a learned positional embedding, KL + per-level BCE loss, and prior-sampling demo. Both examples train on bundled fvdb-example-data meshes in ~1-2 minutes and run headlessly under tests/test_examples.py. Addresses openvdb#741. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
notebooks/04_shape_vae.ipynb trains the shape VAE from examples/shape_vae.py and explores its latent space: reconstructions, prior sampling, batched latent interpolation, and an ipywidgets-based interactive latent explorer (blend between shapes, add prior noise) with a static fallback when ipywidgets is unavailable (e.g. in CI). Executed outputs are stored so the notebook renders on GitHub (reconstruction IoU 0.999; tqdm progress spam stripped). Runs under pytest --nbmake in ~87 s on GPU. Adds ipywidgets to the fvdb_learn environment for the interactive path. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
|
Added |
Switch examples/shape_vae.py and the sampling notebook from the 6 bundled meshes to the 254-model Google Scanned Objects 'Shoe' subset (CC-BY 4.0) now in fvdb-example-data: bump the pinned data revision and add a load_gso_shoes() loader to fvdb.utils.examples. One object category with real intra-class variation (runners, flats, cleats, boots) makes latent interpolation and prior sampling meaningful, mirroring the role of ModelNet40 chairs in MinkowskiEngine's vae.py. Training now runs random minibatches over the pre-voxelized dataset (GridBatch indexing); the ground-truth conv_grid pyramid is built once and sub-indexed per batch. The notebook interpolates a ballet flat into a tall boot and uses a side-on camera. Generative training remains bound by per-iteration generated-topology grid construction (conv_transpose_grid); tracked upstream as openvdb#755. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
|
Dataset upgrade: the shape VAE (example + notebook) now trains on the 254-model Google Scanned Objects Shoe subset (CC-BY 4.0, added to fvdb-example-data in voxel-foundation/fvdb-example-data#1) instead of the 6 bundled meshes — one category with real intra-class variation, mirroring the role of ModelNet40 chairs in ME's CI note: Perf note: generative training is bound by per-iteration generated-topology grid construction ( |
Replace the matplotlib point scatters in the shape-VAE notebook with lit voxel-cube renders: exposed cube faces from pcu.voxel_grid_geometry (interior faces between adjacent voxels culled via duplicate-centroid removal), per-face Lambertian shading, per-panel auto-fit equal-aspect bounds, and unswapped z-up axes (the GSO scans follow the Gazebo z-up convention; the previous y-up swap was showing the shapes sole-on). Shapes are now clearly readable as shoes - laces, soles, and boot shafts are visible in the dataset, reconstruction, sampling, and interpolation figures. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
examples/wip/structure_prediction_net.py targeted a removed API surface (VDBTensor, fvnn.ReLU, fvnn.FillFromGrid, per-module conv backends) and undeclared dependencies (torch_cudamanaged, hardcoded local data paths), so it no longer runs. Its intent - hierarchical structure prediction with per-level occupancy classifiers, losses against a coarsened ground-truth pyramid, U-Net skips onto predicted topology, and a dense neck - is now covered by the maintained, CI-tested examples shape_completion.py and shape_vae.py, which use the generative transposed-convolution path from openvdb#726 plus fvdb.nn.Prune. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Addresses #741.
Summary
fvdb.nn.Prune— theMinkowskiPruninganalog: prunes grid topology and its aligned features by a per-voxel boolean mask (pruned_grid+rmask, differentiable through kept rows). Registered in docs; 13 unit tests.examples/shape_completion.py/examples/shape_vae.py— one-to-one analogs of MinkowskiEngine'scompletion.pyandvae.py, built on the generative transposed convolution from Unify sparse convolution and transpose semantics #726 (from_grid_batch_transposed(..., target_grid=None)) plusPrune.notebooks/04_shape_vae.ipynb— trains the VAE and explores its latent space: reconstructions, prior sampling, latent interpolation (ballet flat → boot), and an optional ipywidgets explorer. Shapes render as lit voxel surfaces.load_gso_shoes()loader and data-revision bump; completion uses the dragon + happy meshes.examples/wip/structure_prediction_net.py(targeted a removed API surface; superseded by the new examples).Design notes
conv_grid(2, 2)pyramid queried withcoords_in_grid— notcoarsened_grid, whose block-centroid lattice differs from whatconv_transpose_grid(2, 2)inverts.keep |= target) and per-level BCE are reproduced verbatim; eval decodes without forcing.Results / verification
pytest tests/unit/test_nn_modules.py: 81 passed (CPU+CUDA, float32/64).pytest tests/test_examples.py -k shape: 2 passed (~4 min, CI runpy path); notebook passespytest --nbmake.black --line-length 120clean.🤖 Generated with Claude Code