Thread through the ability to allow grid objects to own memory

.github/PULL_REQUEST_TEMPLATE.md

@@ -16,6 +16,7 @@ detail.  Why is this change required?  What problem does it solve?-->
 - [ ] Code is formatted. (You can use the format_spiner make target.)
 - [ ] Adds a test for any bugs fixed. Adds tests for new features.
 - [ ] Document any new features, update documentation for changes made.
+- [ ] Update the CHANGELOG.md file.
 - [ ] Make sure the copyright notice on any files you modified is up to date.
 - [ ] LANL employees: make sure tests pass both on the github CI and on the Darwin CI
 - [ ] If ML was used, make sure to add a disclaimer at the top of a file indicating ML was used to assist in generating the file.

CHANGELOG.md

@@ -3,6 +3,7 @@
 ## Current develop
 
 ### Added (new features/APIs/variables/...)
+- [[MR 144]](https://github.com/lanl/spiner/pull/144) Add the ability for grid objects to own memory. This extends the public API but the real changes will come later.
 
 ### Fixed (Repair bugs, etc)
 - [[MR 141]](https://github.com/lanl/spiner/pull/142) fix issue with range() method on device

CMakeLists.txt

@@ -72,7 +72,7 @@ include(cmake/Format.cmake)
 # Add a library
 add_library(spiner INTERFACE)
 add_library(spiner::spiner ALIAS spiner)
-target_compile_features(spiner INTERFACE cxx_std_17)
+target_compile_features(spiner INTERFACE cxx_std_20)
 
 # ##############################################################################
 # Dependencies           #

doc/sphinx/src/databox.rst

@@ -274,6 +274,16 @@ function
 reports how much memory a ``DataBox`` object requires to be externally
 allocated. The function
 
+.. cpp:function:: std::size_t DataBox::dynamicMemorySizeInBytes() const;
+
+reports the size of the value buffer and grid-owned dynamic memory,
+excluding the inline ``DataBox`` object representation. The function
+
+.. cpp:function:: std::size_t DataBox::dumpDynamicMemory(char *dst) const;
+
+writes only that dynamic memory. It is used internally by ``serialize``
+after the inline object bytes have been copied.
+
 .. cpp:function:: std::size_t serialize(char *dst) const;
 
 takes a ``char*`` pointer, assumed to contain enough space for a
@@ -289,10 +299,10 @@ with the overload
 .. cpp:function:: std::size_t DataBox::setPointer(char *src);
 
 sets the underlying tabulated data from the src pointer, which is
-assumed to be the right size and shape. This is useful for the
-deSerialize function (described below) and for building your own
-serialization/de-serialization routines in composite objects. The
-function
+assumed to be the right size and shape, and relocates the trailing
+grid-owned dynamic memory. This is useful for the deSerialize
+function (described below) and for building your own
+serialization/de-serialization routines in composite objects. The function
 
 .. cpp:function:: std::size_t DataBox::deSerialize(char *src);
 
@@ -307,6 +317,16 @@ contained in the ``src`` pointer.
   everything you want to do with the de-serialized ``DataBox`` is
   over.
 
+The serialized memory is laid out as the inline ``DataBox`` metadata,
+followed by the tabulated values and grid-owned dynamic memory. The
+inline grid objects are already part of the ``DataBox`` metadata, so
+their static bytes are not emitted a second time. Dynamic memory is
+dumped for every dimension, including indexed and named dimensions, so
+resource management does not depend on mutable interpolation metadata.
+
+The source buffer must remain alive and suitably aligned for the
+entire lifetime of every ``DataBox`` de-serialized from it.
+
 Putting this all together, an application of
 serialization/de-serialization probably looks like this:
 
@@ -317,7 +337,7 @@ serialization/de-serialization probably looks like this:
   db.loadHDF(filename);
 
   // get size of databox
-  std::size_t allocate_size = db.serialSizeInBytes();
+  std::size_t allocate_size = db.serializedSizeInBytes();
 
   // Allocate the memory for the new databox.
   // In practice this would be an API call for, e.g., shared memory
@@ -335,13 +355,17 @@ serialization/de-serialization probably looks like this:
 
 .. warning::
 
-  The serialization routines described here are **not** architecture
-  aware. Serializing and de-serializing on a single architecture
-  inside a single executable will work fine. However, do not use
-  serialization as a file I/O strategy, as there is no guarantee that
-  the serialized format for a ``DataBox`` on one architecture will be
-  the same as on another. This is due to, for example,
-  architecture-specific differences in endianness and padding.
+  ``DataBox`` serialization is intended for transient communication
+  between processes using the same Spiner version and a compatible
+  architecture. The binary representation is not guaranteed to be
+  compatible across Spiner versions, architectures, compilers, or
+  build configurations. This is due to, for example, differences in
+  endianness, alignment, and padding.
+
+  Do not use serialization as a persistent-storage or web-interchange
+  format. Use HDF5 for persistent, portable storage. Size calculation,
+  serialization, and de-serialization must all use a compatible Spiner
+  build.
 
 .. _`MPI Windows`: https://www.mpi-forum.org/docs/mpi-4.1/mpi41-report/node311.htm
 
@@ -589,3 +613,5 @@ returns the total size of the underlying array in bytes.
 .. cpp:function:: int dim(int i) const;
 
 returns the size in a given dimension/direction, indexed from zero.
+
+Generative AI was used to assist with modifications to this page.

doc/sphinx/src/interpolation.rst

@@ -174,5 +174,31 @@ returns the number of points in the independent variable grid.
 Developer functionality
 ------------------------
 
-For developers, additional functionality is available. Please consult
-the code.
+All grid types implement the resource-management interface used by
+``DataBox``:
+
+.. cpp:function:: std::size_t Grid::serializedSizeInBytes() const;
+.. cpp:function:: std::size_t Grid::dynamicMemorySizeInBytes() const;
+.. cpp:function:: std::size_t Grid::dumpDynamicMemory(char *dst) const;
+.. cpp:function:: std::size_t Grid::serialize(char *dst) const;
+.. cpp:function:: std::size_t Grid::deSerialize(char *src);
+.. cpp:function:: std::size_t Grid::setPointer(char *src);
+.. cpp:function:: Grid Grid::getOnDevice() const;
+.. cpp:function:: void Grid::finalize();
+
+These operations are currently trivial for ``RegularGrid1D`` because
+it contains only inline data. ``PiecewiseGrid1D`` applies them
+recursively to its component grids. This interface allows future grid
+types to own dynamically allocated host or device data without
+requiring grid-specific resource handling in ``DataBox``.
+
+``serialize`` writes the inline grid object followed by its dynamic
+memory. ``dumpDynamicMemory`` writes only the latter, allowing a grid
+embedded in a ``DataBox`` to avoid serializing its inline bytes twice.
+
+The binary serialization methods have the same transient,
+build-dependent compatibility limitations as ``DataBox``
+serialization. See :ref:`the DataBox serialization documentation
+<serialization-and-de-serialization>` for details.
+
+Generative AI was used to assist with modifications to this page.