refactor: replace deepsize crate with custom DeepSizeOf trait for Arrow-aware memory accounting

[wjones127]

The external deepsize crate does not correctly account for Arrow buffers
that are shared across Arc references, causing double-counting in cache
size calculations.

This PR introduces a custom DeepSizeOf trait in lance-core::deepsize
with a Context that tracks both Arc and raw buffer pointers in a
unified HashSet. It also adds a lance-derive proc-macro crate for
#[derive(DeepSizeOf)] and removes the dependency on the external
deepsize crate.

Changes:

• Add lance-core::deepsize module with custom DeepSizeOf trait and pointer-tracking Context
• Add lance-derive proc-macro crate with #[derive(DeepSizeOf)]
• Remove external deepsize crate from all crates; update all impls to use lance_core::deepsize
• Fix all deep_size_of_children implementations to thread context through (previously some ignored the parameter)
• Add DeepSizeOf for arrow_buffer::ScalarBuffer<T> with proper pointer tracking
• Fix PlainPostingList and Positions to use deep_size_of_children with context

[github-actions]

Review

Good motivation — the external deepsize crate doesn't track Arc/buffer pointer identity, so shared Arrow buffers get double-counted in cache size calculations. The custom trait with a pointer-tracking Context is the right approach.

Issues to address

P0: dyn Array::deep_size_of_children calls self.to_data() which clones all buffers

Array::to_data() can be expensive — it materializes an ArrayData by cloning Arc<Buffer>s and for some array types does non-trivial work. This is called every time you measure a column's size in a RecordBatch, which happens on every cache insertion/eviction. Consider using Array::get_array_memory_size() for the total and only using the Context for dedup at the Arc<dyn Array> / Arc<RecordBatch> level. Alternatively, use array.to_data() only once and cache the result, or see if ArrayData can be obtained without cloning (e.g. array.into_data() if you own it, or accessing internal data references).

P1: CompressedPostingList still uses get_buffer_memory_size() without context tracking

CompressedPostingList::deep_size_of_children (rust/lance-index/src/scalar/inverted/index.rs) was only half-migrated — it still calls self.blocks.get_buffer_memory_size() and self.positions.as_ref().map(Array::get_buffer_memory_size) directly, bypassing the Context dedup. This means shared buffers in compressed posting lists can still be double-counted.

P1: ScalarBuffer reports len * size_of::<T>() instead of capacity

In deepsize.rs, ScalarBuffer::deep_size_of_children returns self.len() * size_of::<T>(), but the underlying Buffer may have been allocated with more capacity (e.g. after slicing). Other impls in the same file (like Vec) correctly use capacity. For consistency and accuracy, consider using the underlying buffer's capacity.

Minor

• The HashMap size estimate uses capacity * (size_of::<K>() + size_of::<V>() + 1), which is a reasonable approximation for hashbrown/Swiss tables but the actual overhead per bucket is closer to 1 byte of control metadata per slot (not per occupied entry). This is fine as an estimate since you're using capacity() which already accounts for the total slot count — just noting it's approximate.

• No tests for the derive macro on enums or generics. Consider adding at least one test struct/enum in deepsize.rs tests that uses #[derive(DeepSizeOf)] to verify the proc macro generates correct code.

[codecov]

Codecov Report

❌ Patch coverage is 83.33333% with 67 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-core/src/deepsize.rs	84.34%	35 Missing and 1 partial ⚠️
rust/lance-derive/src/lib.rs	92.53%	4 Missing and 1 partial ⚠️
rust/lance-index/src/scalar/fmindex.rs	0.00%	2 Missing ⚠️
rust/lance/src/index/vector/fixture_test.rs	0.00%	2 Missing ⚠️
rust/lance/src/index/vector/pq.rs	81.81%	2 Missing ⚠️
rust/lance-core/src/container/list.rs	0.00%	1 Missing ⚠️
rust/lance-index/src/scalar/bitmap.rs	50.00%	1 Missing ⚠️
rust/lance-index/src/scalar/btree.rs	66.66%	1 Missing ⚠️
rust/lance-index/src/scalar/inverted/builder.rs	0.00%	1 Missing ⚠️
rust/lance-index/src/scalar/inverted/index.rs	88.88%	1 Missing ⚠️
... and 15 more

📢 Thoughts on this report? Let us know!

[westonpace]

This is nice, I didn't look too closely at all the changes as I think most were find/replace.

Should we do some blanket impl like?

impl<T: lance_core::deepsize::DeepSizeOf> deepsize::DeepSizeOf for T {
  ...
}

Although maybe better to keep the dependency away and let callers do that themselves if they are using deepsize I suppose.

[westonpace]

Are there tests for this macro somewhere? I guess we get some testing via usage? You can, in theory, do integration tests with proc-macros if I recall (unit tests don't make a ton of sense)

Maybe not important for an internal utility.

[wjones127]

You are right, I should add some more tests.

[wjones127]

This is nice, I didn't look too closely at all the changes as I think most were find/replace.

The critical change from deepsize crate is that they track just Arc and RC pointers in Context: ¹:

pub struct Context {
    /// A set of all [`Arc`](std::sync::Arc)s that have already been counted
    arcs: GenericSet<usize>,
    /// A set of all [`Rc`](std::sync::Arc)s that have already been counted
    rcs: GenericSet<usize>,
}

We instead just track one list of pointers:

pub struct Context {
    seen: HashSet<usize>,
}

This is important because Arrow doesn't give you access to the Arc, just the underlying pointers. So if you want to mark Arrow buffers as seen, you can't use the Context from deepsize. (I initially tried to implement deepsize::DeepSizeOf for a new type wrapper around Arrow types, but ran into this. That's what motivated me to make this change.)

Should we do some blanket impl like?

impl<T: lance_core::deepsize::DeepSizeOf> deepsize::DeepSizeOf for T {
  ...
}

I think this might be difficult, given I would need to create some sort of adapter between our Context and their Context. Plus, I don't get the sense that deepsize crate is actively maintained. All the PRs are left as stale. It's our only dependency that still uses syn 1.x. So I lean towards dropping it.

Footnotes

1. https://github.com/Aeledfyr/deepsize/blob/5deebe687695610e6038dc70753defbe7634d549/src/lib.rs#L151-L156 ↩

[westonpace]

I suppose this is one way to work around the orphan rule 😆

[wjones127]

I suppose this is one way to work around the orphan rule 😆

DeepSizeOf is basically unmaintained these days. My PR to upgrade dependencies has been sitting for months with no response. Aeledfyr/deepsize#42