feat(query): minimize deserialize rhs in bitmap functions

[harry-hao]

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

• fixes: Feature: Implement HybridBitmap deserialization calculation based on RoaringFormatSpec #19101

Currently, scalar bitmap AND/OR/XOR/NOT deserialize both operands, and
aggregate OR/XOR/NOT deserialize the rhs, into intermediate treemaps
before operating. This PR minimizes deserialization and operates the
serialized rhs directly to improve performance:

• deps:
  • roaring is patched to a fork revision until upstream releases the required APIs (see "Upstream PR status" below).
• io:
  • HybridBitmap::apply_rhs dispatches (Or|Xor|Sub, SerializedLarge) to new *_assign_serialized_large APIs
  • The intersection wrapper is also rewritten to use *_assign_serialized_large
  • *_assign_serialized_large update the treemap in place instead of rebuilding it
  • avoid deserializing the rhs in the (small lhs, large serialized rhs) case of bitand_assign_serialized_large and sub_assign_serialized_large
• functions:
  • bitmap_and/or/xor/not materialize one side and pass the other as BitmapRhs::Serialized
  • picks the materialized side by operand size (smaller for And, larger for Or/Xor)
• bench:
  • bitmap_op_large scalar pairwise op bench with large workload
  • bitmap_op_mixed scalar pairwise op bench with mixed workload
  • bitmap_not_count aggregate bench

Upstream PR status

RoaringBitmap/roaring-rs#361
is ready for review, which provides:

• 3 borrowing APIs (intersection already exists)
• 4 assigning APIs
• treemap entry API which allows maintaining the treemap instead of rebuilding it

This PR will be marked ready for review after the upstream PR is released.

Performance

The bench commits sit before the first perf commit, so checking out the last bench commit gives the before numbers and HEAD gives the after numbers.

Scalar functions:

op / shape	before	after	change
and high-overlap	23.19 ms	12.85 ms	−45%
and low-overlap	17.95 ms	8.76 ms	−51%
and lhs-larger	42.77 ms	6.62 ms	−85%
and rhs-larger	41.68 ms	6.93 ms	−83%
or high-overlap	480.6 ms	459.1 ms	−4%
or low-overlap	90.33 ms	82.12 ms	−9%
or lhs-larger	314.6 ms	292.0 ms	−7%
or rhs-larger	311.8 ms	285.5 ms	−8%
xor high-overlap	32.89 ms	28.49 ms	−13%
xor low-overlap	27.59 ms	26.25 ms	−5%
xor lhs-larger	61.90 ms	59.03 ms	−5%
xor rhs-larger	63.90 ms	58.35 ms	−9%
not high-overlap	27.05 ms	18.96 ms	−30%
not low-overlap	22.68 ms	14.11 ms	−38%
not lhs-larger	61.05 ms	55.82 ms	−9%
not rhs-larger	40.96 ms	9.28 ms	−77%

• and improves most (−45% to −85%): the new impl skips unmatched rhs containers and always takes the larger side as rhs, while the baseline still deserializes both sides and iterates pairwise
• not improves less (−9% on lhs-larger, −30% to −77% elsewhere): for the same reason as and, it performs better when the rhs is larger
• or/xor improve little (−4% to −13%): they need to read all containers and cost is dominated by serialization (deferring serialization might help)

Aggregates:

workload	before	after	change
bitmap_intersect 1000	5.531 ms	5.256 ms	−5%
bitmap_intersect 65535	349.0 ms	349.6 ms	+0.2%
bitmap_not_count 1000	7.502 ms	1.942 ms	−74%
bitmap_not_count 65535	490.2 ms	121.3 ms	−75%
bitmap_union 1000	282.4 ms	154.0 ms	−45%
bitmap_union 3000	1.766 s	550.9 ms	−69%
bitmap_union 5000	4.434 s	738.6 ms	−83%
bitmap_xor_agg_overlap 1000	180.1 ms	147.5 ms	−18%
bitmap_xor_agg_overlap 3000	1.539 s	1.357 s	−12%
bitmap_xor_agg_overlap 5000	4.514 s	4.087 s	−9%

• bitmap_intersect does not change because it operates on serialized rhs already and its destruct-then-rebuild cost is low, so _assign does not help
• bitmap_not_count improves a lot (−74% to −75%): the new impl skip unmatched rhs containers via difference_assign_with_serialized
• bitmap_union improves with row count (−45% to −83%): new impl avoids len guard in RoaringTreemap::bitor_assign, whose cost grows with row count
• bitmap_xor gains less (−9% to −18%): treemap bitxor_assign has no such length guard, so only the constant cost of the rhs intermediate treemap disappears

Known limitation: support for run containers

This PR upgrades roaring from 0.10.12 to 0.11.5 to get those new APIs. The new version emits run containers when insert_range or optimize is called. Databend does not support run containers, mainly in BitmapReader. This does not impact compatibility, because databend never calls insert_range or optimize, so run containers will not be created. But this can be a potential optimization for future if we support run container in databend.

Tests

• Unit Test
• Logic Test
• Benchmark Test
• No Test - Explain why

Type of change

• Bug Fix (non-breaking change which fixes an issue)
• New Feature (non-breaking change which adds functionality)
• Breaking Change (fix or feature that could cause existing functionality not to work as expected)
• Documentation Update
• Refactoring
• Performance Improvement
• Other (please describe):

AI assistance

• AI usage: An AI coding agent was used to investigate the codebase and draft the new API plumbing.
• Responsible human: @harry-hao
• The responsible human has read every line of this diff and can explain each change

---

This change is 

[harry-hao]

@codex review

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.