dp-lake
Your data is already in S3. Ask it a question.
dp-lake reads Parquet and Apache Iceberg tables on S3 or S3-compatible object storage in place — no warehouse to buy, no loader to run, nothing copied. DuckDB is the engine, the catalog is ours, and the answer comes back through the same pipeline, the same UI, the same agent and the same API as every other query here.
What ships
Exactly this, and nothing more
-
Parquet and Iceberg, on S3 or S3-compatible
Tables in
parquetoricebergformat, addressed bys3://(orfile://for an on-prem volume). S3-compatible stores — MinIO included — work through the datasource's endpoint and URL-style settings. -
Read in place — no warehouse, no loader
Every registered table becomes a view at connect over the objects in your bucket. Nothing is downloaded at registration, nothing is restored into an engine, and there is no load step to keep fresh. The data stays where it is.
-
Read-only, by design
dp-lake is a read connector. There is no write path to the object store — a lake datasource answers questions and never mutates the bucket.
-
DuckDB as the engine
An in-memory DuckDB instance per pooled connection, with operator-set memory, threads and spill limits, and the engine extensions bundled into the published image — a hardened deployment needs no egress to an extension repository.
-
dp-catalog — our own registry
The engine cannot list a bucket, so which tables exist is declared: a small Postgres registry, written over REST or by an agent through the
lake_tables_*MCP tools, and shown read-only on the datasource's page. External catalogs (Glue, S3 Tables, REST) are not read — yet. -
Partition pruning
Hive-partitioned Parquet is scanned with partition pruning: a predicate on the partition column reads only the partitions it names. A date filter is the difference between reading a day and reading two years.
-
Joined to the databases you already run
A lake table is one node in a pipeline. Its rows stage into the execution's in-memory database and join your Postgres, MySQL or SQLite with ordinary SQL — the lake and the operational databases answer one question together.
-
From the UI, from an agent, or as an API
Query it from the browser, let an agent register tables and author the pipeline over MCP, then publish the released pipeline as a governed GET endpoint — the lake becomes an API your applications call.
The demo
Half a billion rideshare trips, read in place
The published demo bucket holds 24 months of NYC TLC high-volume for-hire trips — Uber, Lyft, Via and Juno, 471,851,707 rides — as day-partitioned Parquet, plus a pre-aggregated zone-and-day table, a 1-in-16 sample, an Iceberg copy of that sample and the licence lookup: 855 objects, 7.57 GB in all. Nothing is downloaded at demo start: the lake family has no loader, and a query fetches only the partitions it names. The whole family is documented table by table, licences included, on the demo-data page.
The seeded showcase pipeline nyc/mobility/taxi_vs_rideshare asks one question
across engines: rideshare zone-days from the lake, yellow-taxi trips from Postgres,
the borough lookup from SQLite and the day's rainfall from MySQL — the per-mode rain
lift, rainy days against dry ones, for yellow taxi and for rideshare in each borough (the
taxi side is a hash sample, so within-mode ratios are what stay honest). It ships with
./app.sh --start --demo nyc,trade,lake, which needs egress to S3 (or a mirror
you configure) whenever a lake query runs. The demo's registry is seeded from the published
manifest and holds four tables; the Iceberg copy is published but not registered.
lake_tables_*.| datasource | what it is | access |
|---|---|---|
| sample-lake | Parquet on S3, read in place | read-only ✓ |
| sample-reference | SQLite | read-only ✓ |
from the demo workspace — the agent sees the schema, never the credential
Where this fits
And where it does not
- Egress is real, and the partition column is the lever. A query's bytes cross the network from S3 when the engine scans them. A predicate on the partition column reads only the matching partitions; an unfiltered scan of the trip table reads ~7 GB of objects per run. Write the date filter into the query by default.
- Iceberg tables register by their metadata file. DuckDB 1.5.5.1 cannot resolve a pyiceberg table by its root, so the registry names the table's current metadata file — and a table that still receives commits is re-registered to follow it.
- No scheduler, no dashboards. A pipeline runs when you run it, or when something calls its published endpoint. Recurring cadence and a dashboard surface are not shipped — nothing on this page should read as a promise of either.
- Aggregate in the lake, not in the cursor. The full trip table is 471.9 million rows behind a result cap. Push the aggregation into the lake node — that is what the engine and the pruning are for — and let the pipeline carry the answer, not the raw rows.
Federated query → · Datasources specification → · Run the demo →
Asked before
dp-lake, in three questions
What is dp-lake?
A way to query Parquet and Apache Iceberg tables on S3 or S3-compatible object storage in place: no warehouse, no load step, read-only. DuckDB is the engine, and the catalog is the server's own registry, dp-catalog, because the engine cannot list a bucket - tables are registered by REST or over MCP. See docs/datasources.md section 8C.
Does dp-lake copy my data out of S3?
No. Every registered table becomes a view at connect over the objects in your bucket, and the engine reads only what a query's predicates name - a filter on the partition column reads only the matching partitions. Nothing is downloaded at registration or at demo start; reads are in place, at query time. See docs/datasources.md section 8C.2.
How do I register an Iceberg table in dp-lake?
By its current metadata file - s3://bucket/table/metadata/00042-<uuid>.metadata.json - not by the table root: DuckDB 1.5.5.1 cannot resolve a pyiceberg table's root, so the registry row names the file, and you re-register when the table commits. See docs/datasources.md section 8C.7.