Parent: #27
Depends on benchmark signal from the DuckLake benchmark harness.
Goal
Add an experimental DuckLake destination/sink behind --destination=ducklake.
Proposed approach
Use DuckDB's ducklake extension as the writer instead of implementing DuckLake metadata directly.
Example shape:
INSTALL ducklake;
LOAD ducklake;
ATTACH 'ducklake:<metadata-catalog>' AS streambed (DATA_PATH '<data-path>');
USE streambed;
Scope
- Add destination configuration, e.g.
--destination=iceberg|ducklake.
- Add a DuckLake writer/sink package.
- Create/update schemas and tables through DuckDB SQL.
- Apply flushed CDC batches through SQL operations:
INSERT for insertsDELETE ... USING for deletes- update semantics as delete + insert or SQL
UPDATE, depending on benchmark findings
- Persist/derive
last_flush_lsn semantics for restart safety.
- Keep Iceberg as the default destination.
Open questions
- Best metadata catalog for local/dev and production: DuckDB file vs Postgres catalog.
- How to store Streambed-specific state such as last flushed LSN.
- How much dedup/CoW logic remains in Streambed vs delegated to DuckLake/DuckDB SQL.
Parent: #27
Depends on benchmark signal from the DuckLake benchmark harness.
Goal
Add an experimental DuckLake destination/sink behind
--destination=ducklake.Proposed approach
Use DuckDB's
ducklakeextension as the writer instead of implementing DuckLake metadata directly.Example shape:
Scope
--destination=iceberg|ducklake.INSERTfor insertsDELETE ... USINGfor deletesUPDATE, depending on benchmark findingslast_flush_lsnsemantics for restart safety.Open questions