Generative Transposed Convolution

[lishiyu005]

Dear developers,

I greatly appreciate this good work.

Do you provide 3D generative transposed convolution operation that allows to create new possible coordinates as in Minkowski Engine: https://github.com/NVIDIA/MinkowskiEngine/blob/b71caef9b45ab32c02d2c1b5f48f13c1eb5df526/MinkowskiEngine/MinkowskiConvolution.py#L539

If not do you plan to add a shape completion task similar to https://github.com/NVIDIA/MinkowskiEngine/blob/b71caef9b45ab32c02d2c1b5f48f13c1eb5df526/examples/completion.py?

Thank you very much.

[swahtz]

Hi @lishiyu005 ,

Thanks very much, we're glad that you find it useful!

Yes, we do supply functionality that is equivalent to the MinkowskiEngine generative transposed convolution idea. In fact, fixing up some issues in this functionality was part of the recent PR #726 and will be in the imminent v0.6 release, so I'd recommend you build from source or install nightly builds to try this out.

ConvolutionPlan.from_grid_batch_transposed(kernel_size, stride, source_grid, target_grid=None) selects the COMPLETE topology policy and builds the target as source_grid.conv_transpose_grid(kernel_size, stride). New voxels are generated everywhere any kernel tap can reach, which is the "generative" behavior .

The target grid's voxels can be masked by building a GridBatch from a masked ijk by gathering features with inject_from. Those masked features can be a classifier of which voxels to mask off and can be differentiable. However, we don't have a one-call layer to do this, though it would be very useful… in fact we had examples that did this for the older API as part of a shape completion example but they're old enough now that they don't structurally work but the idea is the same.

Let me go and write this layer and integrate it into an example, as you suggest, since the logic for doing this is common to these types of problems but also perhaps a bit tricky to do right. I can then add it to our nn module and as an example script which will update our old structure prediction example.

[swahtz]

Hi @lishiyu005 , I have put up a PR #753 wrapping the pruning functionality into an easy-to-use nn layer as well as added examples for a shape completion task and a VAE (which has a similar use-case) which will hopefully demonstrate what I was describing. Let me know what you think, if there's anything unclear there

[lishiyu005]

Dear @swahtz ,

Thank you for your great work and updates. Now you added pruning functions which makes the shape completion work. Everything is very clear now.

A question I remained is that for the predicted_grid and gt_grid, are there any easy ways to find corresponding between them? For example first grid in predicted_grid corresponds to second grid in gt_grid.

Best regards