Add DuckLake target support - #75
Conversation
|
Current performance is not good enough. We could try DuckDB as the catalog store (instead of sqlite) |
|
Benchmark update from this session: I added/ran a more realistic lakehouse benchmark matrix comparing Iceberg COW vs DuckLake with SQLite catalog vs DuckLake with DuckDB catalog. Important benchmark shape:
100k baseline rows — write ms, objects in parens: 1M baseline rows — write ms, objects in parens: For 1M + Example read medians for 1M baseline, Takeaway: with realistic small CDC flush sizes, DuckLake is dramatically faster for mutation-heavy workloads and generally has better read latencies through DuckDB. DuckDB catalog is usually a bit faster than SQLite catalog, especially on point/range reads and commit-heavy workloads. Benchmark artifacts from this run were written locally to:
The benchmark harness itself is in |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fb3f4b84bc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return err | ||
| } | ||
| } | ||
| if rows > 0 { | ||
| if err := w.stageRowsAndInsert(ctx, conn, buf); err != nil { |
There was a problem hiding this comment.
When a key is inserted or updated and then deleted before the buffer flushes, the replacement remains in buf.Rows while its key is also placed in buf.Deletes. This flush applies every delete first and then inserts every buffered row, so the deleted replacement is reinserted and remains visible; primary-key chains such as A→B→C similarly retain intermediate rows. Track net-deleted keys and suppress superseded buffered rows before inserting.
Useful? React with 👍 / 👎.
| if !exists { | ||
| w.logger.Info("ducklake truncate on not-yet-created table, no-op", "table", event.Schema+"."+event.Table) | ||
| return nil |
There was a problem hiding this comment.
If a newly discovered table receives inserts and then a TRUNCATE before its first threshold or interval flush, the DuckLake table does not exist yet and this branch returns without clearing the in-memory buffer. The next flush therefore creates the table and writes all pre-TRUNCATE rows back into it. Discard the table's pending rows and deletes before the existence check, as is done for an already-created table.
Useful? React with 👍 / 👎.
| for _, col := range columns { | ||
| if existing[col.Name] { | ||
| continue | ||
| } | ||
| stmt := fmt.Sprintf("ALTER TABLE %s ADD COLUMN %s %s", tableName, quoteIdent(col.Name), pgOIDToDuckDBType(col.OID)) |
There was a problem hiding this comment.
When a PostgreSQL column is dropped or type-changed while the daemon is stopped, the decoder's first relation message has no prior schema and therefore does not invoke HandleSchemaChange; this restart reconciliation path then treats every same-named column as valid and only adds missing columns. Dropped columns consequently remain query-visible, while changed columns retain the old DuckLake type and can coerce incorrectly or make later appends fail (for example, INTEGER→BIGINT once values exceed the old range). Compare existing types and remove columns absent from the incoming schema here as well.
Useful? React with 👍 / 👎.
1 participant
Summary
Adds DuckLake as an additive target format via
--target-format=ducklake, using DuckDB'sducklakeextension with a separate SQLite catalog DB. Iceberg remains the default and its implementation path stays separate.This PR also includes a DuckLake writer performance pass after initial benchmarking showed most latency came from row-by-row staging through
database/sql.What Changed
--target-format=iceberg|ducklake--ducklake-catalog--ducklake-data-pathinternal/ducklakewriter/resync implementation backed by DuckDB extensions:ducklake,sqlite,httpfs.DELETE ... USING, then staged rows +INSERT ... SELECT.QueryAppenderinstead of one SQL insert per staged row.FlushAllacross multiple DuckLake tables into one transaction while preserving per-table resume LSN metadata.Benchmark Notes
These are writer microbenchmarks, not full pipeline benchmarks. Iceberg uses Streambed's handwritten Iceberg writer with in-memory object storage. DuckLake uses DuckDB's DuckLake extension with a local SQLite catalog and local data path. The goal is to compare writer-path overhead and verify real performance improvements without changing benchmark shape.
DuckLake before vs after optimization:
Interpretation: after switching to DuckDB appenders, row-count sensitivity mostly disappears. Remaining latency is dominated by fixed DuckLake transaction/catalog/snapshot work and per-table DML.
Validation
go test ./config ./internal/...passed.go test ./internal/ducklake -count=1 -vpassed.go test -run '^$' -bench 'BenchmarkWriterOperationMatrix' -benchtime=1x -count=5 ./internal/ducklakepassed.go test -run '^$' -bench 'BenchmarkWriterOperationMatrix' -benchtime=1x -count=3 ./internal/icebergpassed.go build ./cmd/streambedpassed, with a non-fatal sandbox stat-cache warning writing under the normal Go module cache.Earlier in the session, focused Docker-backed DuckLake integration tests passed before the performance pass. After the performance pass, Docker daemon access was blocked by the sandbox, so I could not rerun those integration tests locally.
Known baseline note: the broader existing integration suite has two Iceberg failures on
origin/mainas well:TestFlushThresholdCreatesExpectedParquetAndMetadataandTestPostgresTypeRoundTripToIceberg.