feat(query): add page index by zhang2014 · Pull Request #20079 · databendlabs/databend

zhang2014 · 2026-06-29T03:44:57Z

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

feat(query): add page index

    flowchart LR
        S[Snapshot] --> SG[Segment]
        SG --> BM[BlockMeta]

        BM --> D["_b/block.parquet<br/>Data pages by granule"]
        BM --> M["_i_p/mins.parquet<br/>m{i}: cluster-key min"]
        BM --> O["_i_p/offsets.parquet<br/>g_{column_id}: data page offsets<br/>gbloom_*: Bloom payload ranges"]

        O --> D
        O --> B["_i_gb/index/version/block_col.gbloom<br/>One Bloom page per granule"]

        style M stroke-dasharray: 5 5

Tests

Unit Test
Logic Test
Benchmark Test
No Test - Explain why

Type of change

Bug Fix (non-breaking change which fixes an issue)
New Feature (non-breaking change which adds functionality)
Breaking Change (fix or feature that could cause existing functionality not to work as expected)
Documentation Update
Refactoring
Performance Improvement
Other (please describe):

This change is

…uning Introduce a ClickHouse-style sparse page index that maps cluster-key granules to physical page byte ranges, enabling sub-block pruning on the read path. Write path: - Add PageIndexWriter that builds a columnar parquet sidecar file with per-granule cluster-key min values and leaf-column page offsets. - Integrate granule splitting into the stream block builder based on the new index_granularity table option. - Extend WriteSettings, BlockWriter, and location helpers to produce and persist the page index file. Read path: - Add SparsePageIndexPruner that evaluates predicates against granule cluster-key ranges and narrows column reads to relevant byte ranges. - Extend FusePart and ReadState to carry page-level byte ranges. - Wire page-index-aware merge IO into block_reader_merge_io_async. Pruning: - Remove the old page_pruner module from common/pruner (superseded). - Integrate sparse page index pruning into the fuse block pruner. Validation & tests: - Add index_granularity to table option validation. - Add sqllogictest covering end-to-end write and read with the index. - Update existing tests to accommodate new BlockMeta/parquet fields.

Add fuse compatibility test cases to ensure: - Forward compat: current writer with page-index sidecar files and new BlockMeta fields is readable by older readers that ignore the unknown sidecar and fall back to whole-block reads. - Backward compat: old writer with inline ClusterStatistics.pages (row_per_page) is still readable by the current reader after removal of the legacy page_pruner module. Both suites cover point lookups, range predicates, out-of-range predicates, and full scans over single and multi-column cluster keys.

CI lint (check_macos / check) failed on test_cases.yaml:40 with "too many blank lines (1 > 0)". Strip the extra trailing blank. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

The sqllogictest column-type markers (e.g. `query IIT`) are flagged by typos as misspellings of `IT`. tests/sqllogictests is already excluded for the same reason; apply the same exemption to the compat-logictest suites. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

CI `cargo clippy -D warnings` failed with clippy::type_complexity on ArrowParquetWriter::finish's 3-tuple return. Introduce a named FinishedRowGroup struct for the row-group result. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

github-actions · 2026-06-29T07:10:36Z

Docker Image for PR

tag: pr-20079-db56cca-1782716858

note: this image tag is only available for internal use.

# Conflicts: # src/query/service/tests/it/indexes/spatial_index/runtime_filter.rs # src/query/service/tests/it/storages/fuse/bloom_index_meta_size.rs # src/query/service/tests/it/storages/fuse/operations/prewhere.rs # src/query/storages/common/cache/src/manager.rs # src/query/storages/common/index/src/page_index.rs # src/query/storages/common/pruner/src/page_pruner.rs # src/query/storages/common/table_meta/src/meta/v2/segment.rs # src/query/storages/common/table_meta/src/meta/v3/frozen/block_meta.rs # src/query/storages/fuse/src/fuse_part.rs # src/query/storages/fuse/src/io/write/block_writer.rs # src/query/storages/fuse/src/io/write/stream/block_builder.rs # src/query/storages/fuse/src/operations/read/read_block_context.rs # src/query/storages/fuse/src/operations/read_partitions.rs # src/query/storages/fuse/src/pruning/expr_runtime_pruner.rs # src/query/storages/fuse/src/pruning_pipeline/column_oriented_block_prune.rs # src/query/storages/fuse/src/statistics/cluster_statistics.rs

…e_index

numbers(5000, 5000) is invalid — numbers() accepts only a row count. Use `number + 5000 FROM numbers(5000)` to insert the second batch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

The new BLOOM token adds itself to the "expecting" keyword list in DROP parse errors; regenerate the goldenfile to match. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

# Conflicts: # src/query/storages/common/index/src/granule_index.rs # src/query/storages/common/index/tests/it/page_pruner.rs # src/query/storages/common/index/tests/it/testdata/test_page_indies.txt # src/query/storages/common/io/src/merge_io_reader.rs # src/query/storages/common/pruner/src/block_meta.rs # src/query/storages/common/pruner/src/topn_pruner.rs # src/query/storages/fuse/src/io/write/block_writer.rs # src/query/storages/fuse/src/io/write/stream/block_builder.rs # src/query/storages/fuse/src/operations/changes.rs # src/query/storages/fuse/src/pruning/block_pruner.rs # src/query/storages/fuse/src/pruning/fuse_pruner.rs # src/query/storages/fuse/src/pruning_pipeline/column_oriented_block_prune.rs

…refactor/page_index # Conflicts: # src/query/service/src/physical_plans/physical_recluster.rs

github-actions · 2026-08-24T19:04:19Z

Docker Image for PR

tag: pr-20079-6ce4104-1787598088

note: this image tag is only available for internal use.

…refactor/page_index

github-actions · 2026-08-25T17:02:11Z

Docker Image for PR

tag: pr-20079-47acbea-1787677157

note: this image tag is only available for internal use.

…refactor/page_index # Conflicts: # src/query/ast/tests/it/testdata/stmt-error.txt

github-actions · 2026-08-27T18:09:53Z

Docker Image for PR

tag: pr-20079-0cb210a-1787854010

note: this image tag is only available for internal use.

github-actions · 2026-08-27T21:07:43Z

Docker Image for PR

tag: pr-20079-145af32-1787864687

note: this image tag is only available for internal use.

github-actions · 2026-08-30T11:46:46Z

Docker Image for PR

tag: pr-20079-a471870-1788090236

note: this image tag is only available for internal use.

github-actions · 2026-08-30T15:48:55Z

Docker Image for PR

tag: pr-20079-4c14950-1788104640

note: this image tag is only available for internal use.

zhang2014 added 2 commits June 29, 2026 09:01

github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Jun 29, 2026

zhang2014 and others added 3 commits June 29, 2026 12:00

fix(compat): remove trailing blank line failing yamllint

8e7dc5a

CI lint (check_macos / check) failed on test_cases.yaml:40 with "too many blank lines (1 > 0)". Strip the extra trailing blank. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

zhang2014 added the ci-cloud Build docker image for cloud test label Jun 29, 2026

zhang2014 and others added 22 commits July 3, 2026 17:50

z

e1dfdbe

z

5d8b3c0

z

32f40df

Merge remote-tracking branch 'commit_upstream/main' into refactor/pag…

a657552

…e_index

z

83e24a0

fix: numbers() takes one arg in page bloom index test

37e3f62

numbers(5000, 5000) is invalid — numbers() accepts only a row count. Use `number + 5000 FROM numbers(5000)` to insert the second batch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

test: regenerate stmt-error golden for new BLOOM keyword

f4ad499

The new BLOOM token adds itself to the "expecting" keyword list in DROP parse errors; regenerate the goldenfile to match. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

z

a797dd3

z

3d128d6

z

72327b5

z

01984fe

z

2d18aba

z

b519327

z

906b668

z

9e5b700

z

f0a14f4

z

ef7cd37

z

92691a7

z

26ddf95

Merge remote-tracking branch 'upstream/main' into refactor/page_index

31c6a18

zhang2014 added 2 commits August 25, 2026 00:16

Merge branch 'main' of https://github.com/datafuselabs/databend into …

46a00c3

…refactor/page_index # Conflicts: # src/query/service/src/physical_plans/physical_recluster.rs

z

c347621

zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 24, 2026

zhang2014 added 2 commits August 25, 2026 03:22

fix(query): align tests with table block thresholds

20c2e1c

Merge branch 'main' of https://github.com/datafuselabs/databend into …

f82889b

…refactor/page_index

zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 25, 2026

zhang2014 added 2 commits August 27, 2026 22:27

Merge branch 'main' of https://github.com/datafuselabs/databend into …

587f952

…refactor/page_index # Conflicts: # src/query/ast/tests/it/testdata/stmt-error.txt

z

a2f8dde

zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 27, 2026

z

e86d4e7

zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 27, 2026

z

5cc8b9f

zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 30, 2026

z

9ef547d

zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 30, 2026

z

d989837

zhang2014 added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Aug 30, 2026

Navigation Menu

Uh oh!

feat(query): add page index - #20079

feat(query): add page index#20079
zhang2014 wants to merge 81 commits into
databendlabs:mainfrom
zhang2014:refactor/page_index

zhang2014 commented Jun 29, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 29, 2026

Uh oh!

github-actions Bot commented Aug 24, 2026

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

github-actions Bot commented Aug 27, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

zhang2014 commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Tests

Type of change

Uh oh!

github-actions Bot commented Jun 29, 2026

Docker Image for PR

Uh oh!

github-actions Bot commented Aug 24, 2026

Docker Image for PR

Uh oh!

github-actions Bot commented Aug 25, 2026

Docker Image for PR

Uh oh!

github-actions Bot commented Aug 27, 2026

Docker Image for PR

Uh oh!

github-actions Bot commented Aug 27, 2026

Docker Image for PR

Uh oh!

github-actions Bot commented Aug 30, 2026

Docker Image for PR

Uh oh!

github-actions Bot commented Aug 30, 2026

Docker Image for PR

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhang2014 commented Jun 29, 2026 •

edited

Loading