Codec Framework and Delayed Loading Removal

[danrbailey]

This introduces the new codec framework and completely removes delayed-loading / out-of-core into the master branch.

All of the components included in this PR have been reviewed independently:

#2155
#2164
#2166
#2179
#2180
#2184
#2185
#2197
#2196
#2230

These changes have been tested extensively at ILM.

For completeness, here are the new additions to the changelog on this branch from aggregating all the pending changes:

OpenVDB:
    Highlights:
    - Completely removed support for out-of-core / delayed loading across OpenVDB
      core, including removal of the OPENVDB_USE_DELAYED_LOADING build option,
      DelayedLoadMetadata, and TempFile. This was done to improve performance,
      reduce codebase complexity and to entirely eliminate the optional Boost
      dependency.
    New features:
    - Introduced new openvdb::codecs subsystem and CodecRegistry for stream
      encoding and decoding.
    - Added support for implicit codec conversion fallback (e.g. float to half,
      scalar to mask) and ReadDiagnostics.
    API changes:
    - Extracted Point Data I/O function overloads from PointDataGrid.h into new
      openvdb/points/PointDataIO.h header.
    - GridDescriptor::read() has been deprecated, use GridDescriptor::readHeader()
      followed by GridDescriptor::readStreamPos() instead.
    - Added optional trailing WriteOptions parameter to io::File::write() and
      io::Stream::write(). No changes in behavior.
    - Added optional trailing ReadOptions parameter to io::File::getGrids() and
      io::File::readGrid(). No changes in behavior.
    - TreeBase::readTopology() and TreeBase::writeTopology() are now pure virtual.
      Derived classes of TreeBase now need to read and write int32_t(1) to remain
      backwards-compatible.
    Improvements:
    - Large refactor of I/O classes, many private and protected member functions have
      been modified. No change in behavior for io::File and io::Stream.
    - Remove direct testing of Tree I/O methods in favor of indirect testing using
      the Grid I/O.
    Houdini:
    - Add new Vulkan viewport support for VDB Points primitives and remove support
      for delayed loading.

[apradhana]

This may be blocking because of ABI incompatibilities between 13.0 and 13. 1.

StreamMetadata is shared between coexisting OpenVDB versions and is part of the protected Grid ABI. A 13.1 reader can access or copy this field from an Impl created by 13.0, where this bool does not exist or was never initialized. Tail padding does not make that access safe. Could this flag use existing compatible storage such as AuxDataMap, or be restricted to ABI 14?

For example, setAllocateLeafBuffers(true) could store true under a private key in auxData(). allocateLeafBuffers() could look up that key with std::any_cast<bool>, and clearing the flag could erase it. This uses storage that already exists and is initialized in ABI 13, so no new Impl member is needed.

[apradhana]

From TSC meeting 8/26/2026 this is okay, because ABI changes is applied only for Tree and Grid contracts say that we want to be able to reinterpret_cast the pointer. In the discussion, it was mentioned that it's for the old grid reading workflow. It is used to be read during a topology-only read. So it seems okay.

For completeness, I'll also include a discussion all the way back from 2019: https://github.com/AcademySoftwareFoundation/openvdb/blob/master/tsc/meetings/2019-09-26.md?plain=1#L27-L45. So there may be a compatibility issue with Houdini.

[apradhana]

This is mainly a comment regarding the removal of PImpl and not a blocking issue. I just want us all to be aware that removing PImpl reduces ABI flexibility. With PImpl, private implementation members can be changed without changing the public layout of File or Stream. For example, a new version could add more fields to the hidden implementation while the public object remains just a pointer.

Without PImpl, those private members become part of the public object’s binary layout. Future internal changes may therefore require applications and plugins to be rebuilt. Mixing versions also becomes unsafe: old headers might allocate 72 bytes for File, while the new library expects 296 bytes, potentially causing crashes or memory corruption.

Again, this is non-blocking.

[danrbailey]

This is mainly a comment regarding the removal of PImpl and not a blocking issue. I just want us all to be aware that removing PImpl reduces ABI flexibility. With PImpl, private implementation members can be changed without changing the public layout of File or Stream. For example, a new version could add more fields to the hidden implementation while the public object remains just a pointer.

Without PImpl, those private members become part of the public object’s binary layout. Future internal changes may therefore require applications and plugins to be rebuilt. Mixing versions also becomes unsafe: old headers might allocate 72 bytes for File, while the new library expects 296 bytes, potentially causing crashes or memory corruption.

Again, this is non-blocking.

This is sort of a general comment about why pimpl might be useful. If we remove delayed loading, then the StreamMetadata is no longer part of the Grid ABI because you cannot pass around a partially created grid. Using the codec framework improves flexibility and does away with all the complexity of managing and maintaining StreamMetadata objects. That is the overall intention here.

[apradhana]

This is mainly a comment regarding the removal of PImpl and not a blocking issue. I just want us all to be aware that removing PImpl reduces ABI flexibility. With PImpl, private implementation members can be changed without changing the public layout of File or Stream. For example, a new version could add more fields to the hidden implementation while the public object remains just a pointer.
Without PImpl, those private members become part of the public object’s binary layout. Future internal changes may therefore require applications and plugins to be rebuilt. Mixing versions also becomes unsafe: old headers might allocate 72 bytes for File, while the new library expects 296 bytes, potentially causing crashes or memory corruption.
Again, this is non-blocking.

This is sort of a general comment about why pimpl might be useful. If we remove delayed loading, then the StreamMetadata is no longer part of the Grid ABI because you cannot pass around a partially created grid. Using the codec framework improves flexibility and does away with all the complexity of managing and maintaining StreamMetadata objects. That is the overall intention here.

Thanks for clarifying this and for the discussion during TSC.

[apradhana]

Codecs pointed out: For numeric-to-Mask conversion (ReadMode::Mask), the result should use activity rather than numeric values. This cast, and the tile-value casts below, can make the MaskGrid disagree with the source topology. A nonzero background or inactive 0.1f tile becomes true, while an active zero tile can become false. Could this be special-cased for the Mask conversion so inactive entries and the background are false, and active entries are true?

[danrbailey]

This is a good suggestion, I like this. In fact, I think this should ultimately become a tool that lives in tools/Mask.h and turns a normal numeric grid into a mask grid that works with activity and ignores values. In lieu of doing that right now, I've implemented your suggestion here:

ab58a58

I know you have already approved that PR, but let me know if you have any concerns about that, otherwise I'll merge that into this feature/io branch along with the other changes on that PR.