Skip to content

Subquery (SubPlan) pushdown #289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JoshDreamland wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Subquery (SubPlan) pushdown #289

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d9c6d11
Push down subquery (SubPlan) residue to ClickHouse
JoshDreamland Jul 1, 2026
91306a7
Update TPC-H results
theory Jul 1, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions .github/workflows/clickhouse.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ permissions:

jobs:
build:
env: { pg: 19 }
env: { pg: 19, CH_RELEASE: "${{ matrix.ch }}" }

@theory theory Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for this change?

strategy:
fail-fast: false
matrix:
Expand Down Expand Up @@ -38,7 +38,6 @@ jobs:
uses: actions/checkout@v6
with: { submodules: true }
- name: Start ClickHouse
env: { CH_RELEASE: "${{ matrix.ch }}" }
run: .github/ubuntu/clickhouse.sh
- name: Test
run: pg-build-test
14 changes: 13 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,20 @@ $(EXTENSION)-$(DISTVERSION).zip:
git archive-all -v --prefix "$(EXTENSION)-$(DISTVERSION)/" --force-submodules $(EXTENSION)-$(DISTVERSION).zip

.PHONY: test/schedule # Depends on $(TESTS), so always rebuild.
# The subquery-pushdown tests exercise SubPlan pushdown, which is gated on
# ClickHouse 25.8+ (older analyzers error on the correlated SQL). When CH_RELEASE
# names an older server, drop those tests and emit a GitHub Actions warning
# rather than carry a second set of "not pushed down" expected files. CH_RELEASE
# is unset for local runs and the Postgres matrix (which use the latest CH), so
# they keep the tests.
Comment on lines +74 to +79

@theory theory Jul 1, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This confuses me. CH_RELEASE is currently set in the GitHub workflow but nowhere else. Someone downloading it and running the tests won't have it set. I don't have it set! The Makefile has not and should not have any idea of the ClickHouse version IMO. Nor should it have any awareness of GitHub Actions.

test/schedule:
@echo "test: $(patsubst test/sql/%.sql,%,$(TESTS))" > $@
@tests='$(patsubst test/sql/%.sql,%,$(TESTS))'; \
maj=$${CH_RELEASE%%.*}; rest=$${CH_RELEASE#*.}; min=$${rest%%.*}; \
if [ -n "$$CH_RELEASE" ] && echo "$$maj$$min" | grep -qE '^[0-9]+$$' && [ $$((maj * 100 + min)) -lt 2508 ]; then \
tests=$$(echo "$$tests" | tr ' ' '\n' | grep -vxE 'subplan_pushdown|subquery_eq' | tr '\n' ' '); \
echo "::warning::Skipping subquery-pushdown tests (subplan_pushdown, subquery_eq): they require ClickHouse 25.8+, but this run uses ClickHouse $$CH_RELEASE"; \
fi; \
echo "test: $$tests" > $@

installcheck: test/schedule

Expand Down
50 changes: 25 additions & 25 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,36 +31,36 @@ and pushing down queries.

This table compares [TPC-H] query performance between regular PostgreSQL
tables and pg_clickhouse connected to ClickHouse, both loaded at scaling
factor 1; ✔︎ indicates full pushdown, while a dash indicates a query
cancellation after 1m. All tests run on a MacBook Pro M4 Max with 36 GB of
memory.
factor 1; ✔︎ indicates full pushdown as a single foreign scan and ✼ indicates
full pushdown as multiple foreign scans. All tests run on a MacBook Pro M4 Max
with 36 GB of memory.

<!-- cd dev/tpch && make ch && make pg && make run -->

| Query | PostgreSQL | pg_clickhouse | Pushdown |
| ----------:| ----------:| -------------:|:--------:|
| [Query 1] | 4693 ms | 268 ms | ✔︎ |
| [Query 2] | 458 ms | 3446 ms | |
| [Query 3] | 742 ms | 111 ms | ✔︎ |
| [Query 4] | 270 ms | 130 ms | ✔︎ |
| [Query 5] | 337 ms | 1460 ms | ✔︎ |
| [Query 6] | 764 ms | 53 ms | ✔︎ |
| [Query 7] | 619 ms | 96 ms | ✔︎ |
| [Query 8] | 342 ms | 156 ms | ✔︎ |
| [Query 9] | 3094 ms | 298 ms | ✔︎ |
| [Query 10] | 581 ms | 197 ms | ✔︎ |
| [Query 11] | 212 ms | 24 ms | |
| [Query 12] | 1116 ms | 84 ms | ✔︎ |
| [Query 13] | 958 ms | 1368 ms | |
| [Query 14] | 181 ms | 73 ms | ✔︎ |
| [Query 15] | 1118 ms | 557 ms | |
| [Query 16] | 497 ms | 1714 ms | |
| [Query 17] | 1846 ms | 32709 ms | |
| [Query 18] | 5823 ms | 10649 ms | |
| [Query 19] | 53 ms | 206 ms | ✔︎ |
| [Query 20] | 421 ms | - | |
| [Query 21] | 1349 ms | 4434 ms | |
| [Query 22] | 258 ms | 1415 ms | |
| [Query 1] | 4483 ms | 59 ms | ✔ |
| [Query 2] | 588 ms | 24 ms | ✔ |
| [Query 3] | 786 ms | 62 ms | ✔ |
| [Query 4] | 550 ms | 38 ms | ✔ |
| [Query 5] | 721 ms | 1439 ms | ✔ |
| [Query 6] | 592 ms | 17 ms | ✔ |
| [Query 7] | 639 ms | 29 ms | ✔ |
| [Query 8] | 398 ms | 383 ms | ✔ |
| [Query 9] | 2842 ms | 162 ms | ✔ |
| [Query 10] | 860 ms | 125 ms | ✔ |
| [Query 11] | 276 ms | 21 ms | ✼ |
| [Query 12] | 963 ms | 26 ms | ✔ |
| [Query 13] | 1037 ms | 1354 ms | |
| [Query 14] | 675 ms | 30 ms | ✔ |
| [Query 15] | 2520 ms | 387 ms | |
| [Query 16] | 539 ms | 823 ms | |
| [Query 17] | 2107 ms | 37 ms | ✔ |
| [Query 18] | 5230 ms | 7228 ms | |
| [Query 19] | 68 ms | 47 ms | ✔ |
| [Query 20] | 473 ms | 28 ms | |
| [Query 21] | 1145 ms | 4470 ms | |
| [Query 22] | 270 ms | 45 ms | ✼ |

### Compile From Source

Expand Down
49 changes: 25 additions & 24 deletions dev/tpch/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,34 +8,35 @@ PostgreSQL performance to pg_clickhouse performance.
The scripts run each query in [queries](queries) three times each for native
PostgreSQL and pg_clickhouse performance and produces a Markdown table
reporting the averaged times for each, as well as whether the pg_clickhouse
query pushed down to ClickHouse. Times exceeding 60s will not be recorded, and
result in a `-`. An example:
query fully pushed down to ClickHouse as a single query (`✔`) or multiple
queries (`✼`). Times exceeding 60s will not be recorded, and result in a `-`.
An example:

```md
| Query | PostgreSQL | pg_clickhouse | Pushdown |
| ----------:| ----------:| -------------:|:--------:|
| [Query 1] | 4693 ms | 268 ms | ✔︎ |
| [Query 2] | 458 ms | 3446 ms | |
| [Query 3] | 742 ms | 111 ms | ✔︎ |
| [Query 4] | 270 ms | 130 ms | ✔︎ |
| [Query 5] | 337 ms | 1460 ms | ✔︎ |
| [Query 6] | 764 ms | 53 ms | ✔︎ |
| [Query 7] | 619 ms | 96 ms | ✔︎ |
| [Query 8] | 342 ms | 156 ms | ✔︎ |
| [Query 9] | 3094 ms | 298 ms | ✔︎ |
| [Query 10] | 581 ms | 197 ms | ✔︎ |
| [Query 11] | 212 ms | 24 ms | |
| [Query 12] | 1116 ms | 84 ms | ✔︎ |
| [Query 13] | 958 ms | 1368 ms | |
| [Query 14] | 181 ms | 73 ms | ✔︎ |
| [Query 15] | 1118 ms | 557 ms | |
| [Query 16] | 497 ms | 1714 ms | |
| [Query 17] | 1846 ms | 32709 ms | |
| [Query 18] | 5823 ms | 10649 ms | |
| [Query 19] | 53 ms | 206 ms | ✔︎ |
| [Query 20] | 421 ms | - | |
| [Query 21] | 1349 ms | 4434 ms | |
| [Query 22] | 258 ms | 1415 ms | |
| [Query 1] | 4483 ms | 59 ms | ✔ |
| [Query 2] | 588 ms | 24 ms | ✔ |
| [Query 3] | 786 ms | 62 ms | ✔ |
| [Query 4] | 550 ms | 38 ms | ✔ |
| [Query 5] | 721 ms | 1439 ms | ✔ |
| [Query 6] | 592 ms | 17 ms | ✔ |
| [Query 7] | 639 ms | 29 ms | ✔ |
| [Query 8] | 398 ms | 383 ms | ✔ |
| [Query 9] | 2842 ms | 162 ms | ✔ |
| [Query 10] | 860 ms | 125 ms | ✔ |
| [Query 11] | 276 ms | 21 ms | ✼ |
| [Query 12] | 963 ms | 26 ms | ✔ |
| [Query 13] | 1037 ms | 1354 ms | |
| [Query 14] | 675 ms | 30 ms | ✔ |
| [Query 15] | 2520 ms | 387 ms | |
| [Query 16] | 539 ms | 823 ms | |
| [Query 17] | 2107 ms | 37 ms | ✔ |
| [Query 18] | 5230 ms | 7228 ms | |
| [Query 19] | 68 ms | 47 ms | ✔ |
| [Query 20] | 473 ms | 28 ms | |
| [Query 21] | 1145 ms | 4470 ms | |
| [Query 22] | 270 ms | 45 ms | ✼ |
```

## Setup & Execution
Expand Down
14 changes: 11 additions & 3 deletions dev/tpch/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,17 @@ function print_query {
else
ch=$(psql -c "SELECT round(AVG(x)) || ' ms' FROM unnest(ARRAY[$(join_by ', ' "${ch_times[@]}")]) AS x" --tuples-only --no-psqlrc --quiet --no-align)
fi
# Full pushdown means outer most plan node is a foreign scan, and no inner
# (indented) nodes are foreign scans.
check=$(grep -q '^Foreign Scan' "result/ch$i.1" && ! grep -q ' Foreign Scan' "result/ch$i.1" && printf '✔︎' || printf ' ')
# Full pushdown means outer most plan node is a foreign scan. Record as an
# asterisk if there is a second internal (indented) foreign scan; still
# fully pushed down, but not to a single query.
check='✔'
if grep -q '^Foreign Scan' "result/ch$i.1"; then
if grep -q ' Foreign Scan' "result/ch$i.1"; then
check='✼'
fi
else
check=' '
fi
printf "| %10s | %10s | %13s | %s |\n" "[Query $i]" "$pg" "$ch" "$check"
}

Expand Down
Loading
Loading