Sync the data, or query it where it is
Fivetran and Airbyte are ELT tools: connectors that copy your operational databases into a warehouse on a schedule, so that transformation can happen there. datapipelines is a different shape — every node runs as a query at its source, the join happens in a scratch database created for that one run and dropped after it, and the result is released as a versioned API. No copy, no warehouse, no sync lag. This page is about when that difference matters, and — just as honestly — when it does not.
What each one is for
Fivetran and Airbyte solve a real problem: the data your analytics needs lives in a dozen systems, and the warehouse needs it delivered, on time, in the right shape, every night. Their connectors handle the long tail — hundreds of sources, incremental syncs, retries, schema drift — so the warehouse stays fed without a team writing extract scripts. If the destination is the point, they are the right tools, and the honest part of this page is saying so up front.
datapipelines solves a different one. Most teams' operational questions — the revenue by borough, the
churn by cohort, the stockout risk by store — live in two or three databases, and the question crosses
them. The ELT answer is: copy both into a warehouse first. The datapipelines answer is: run each source's
query where the data lives, stage the results into a scratch database that exists for exactly that run,
join them there with ordinary SQL, and drop the scratch database when the run ends. The pipeline is a
versioned JSON document an agent can draft and a person can release; the result can be published as a
GET endpoint. Between the question and the answer, nothing is copied anywhere permanent.
The vocabulary overlaps, which is why the comparison gets asked for: both products say "pipeline", both say "sync", both promise the numbers without hand-written scripts. The test to apply is one question — where does the work happen? If the honest answer is "in a system built to receive copies", the tool is ELT and proud of it. If the answer is "at the sources, joined in a workbench that vanishes", it is this. The rest of the page is that one answer, priced in the currencies that matter: freshness, surface area, and the step where the number becomes something a program can call.
The cost of a copy
A sync runs on a schedule, so the warehouse answers with the data as of the last load — an hour, a night, a weekend old. A query at the source answers with now. When the consumer is a person refreshing a dashboard that is a preference; when it is an application serving a customer, it is a bug report waiting to happen.
The copy has to live somewhere, and the somewhere is a warehouse: a deployment to run, secure, tune and pay an on-call for — before a single question has been answered. The scratch database here is created when a run starts and destroyed when it ends; there is no standing system between your databases and the answer.
Every copy widens the blast radius: the customers table now exists in the warehouse too, with its own access rules, its own retention, its own things to forget. Answering in place means the PII never leaves the system that already governs it — the agent reads it through a key, the server holds the credentials, and nothing lands.
Say the ELT pipeline finishes. The numbers are in the warehouse. The product team still needs an endpoint that serves them — a service, a deployment, an auth story. Here the release is the API step: a published endpoint serves the released version, keys bind to its path, and nothing else has to be built.
Where datapipelines stops
First, history. The query sees the source as it is now, and the scratch database is dropped after the run — if the source overwrites yesterday, yesterday is gone, and no query at the source will bring it back. History is the ELT tools' home ground; that is one of the reasons they exist.
Second, the warehouse itself. Write-back here means a pipeline node can land its result in a table of a datasource you already run — useful for feeding a BI tool, and explicitly not a columnar warehouse with a modelling layer on top. Third, scheduling: pipelines run on demand, by an agent, a REST call or a person; the scheduler is the planned roadmap item, and scheduled extracts to your own lake are later. The roadmap page labels both as planned and later, undated. Fourth, transformation breadth: the work is SQL, plus JSONata and JavaScript transform nodes that are themselves a roadmap item — if the job is Python operators orchestrating a platform, the Airflow comparison says plainly that Airflow is the answer.
None of these four is a defect; each is a boundary drawn on purpose, and the page would rather you know them before the pilot than after. The pattern that works: teams keep the ELT copy where history and the warehouse genuinely earn their keep, and answer the operational, cross-source, fresh-number questions here — the two coexist, and the FAQ has the "can I use both?" answer in one paragraph.
The short version
What it looks like
demo/top_company_by_borough mid-run — the lake read in place, the join staged, the result in the results dock. No warehouse between them.you › Which rideshare company carried the most trips in each borough last quarter?
| borough | top company | trips | borough trips | share |
|---|---|---|---|---|
| Manhattan | Uber | 17,660,839 | 23,669,163 | 74.62% |
| Brooklyn | Uber | 11,946,362 | 16,057,699 | 74.40% |
| Queens | Uber | 9,786,602 | 12,847,231 | 76.18% |
| Bronx | Uber | 6,078,750 | 7,572,899 | 80.27% |
| Staten Island | Uber | 715,593 | 918,360 | 77.92% |
GET /api/x/demo/top-company-by-borough?anchor_date=2025-01-01 · after release · key scoped to /demo
from the demo workspace — a real result, not a promise
The demo stack runs this pattern end to end: ./app.sh --start --demo nyc,lake brings up
Postgres, MySQL, SQLite and Parquet on S3, and the seeded pipelines — nyc/mobility/revenue_by_borough
joins across engines, nyc/mobility/taxi_vs_rideshare adds the lake — answer questions the ELT
shape would have copied four sources to reach.
Asked before
No. Every node runs as a query at its source and the result is staged into a scratch database created for that one run, joined there, and dropped when the run ends. Nothing is landed into a warehouse. How staging works is docs/staging.md §3.
Not the copy itself — populating a warehouse is the job an ELT tool does, and the honest page says so. What replaces the nightly script is the pipeline: the join an agent drafts, a person releases, and an API serves; write-back can also land a result in one table you already have. Publishing is docs/rest-api.md §19.
This does not keep history: the query sees the source as it is now, and the per-run scratch database is dropped after the run. A scheduled extract to your own lake — where that history would land — is a later roadmap item, and the roadmap page labels it as such. The roadmap is docs/ROADMAP.md §3.
Yes, and it is a common shape: datapipelines joins the operational sources and serves the result as a versioned GET endpoint, while your ELT tool keeps feeding the warehouse the wide history it is good at. The endpoint contract is docs/rest-api.md §19.
The demo proves the shape on a laptop; your sources prove it for real. Register the databases, let the agent draft the join, release it when the numbers are the numbers — and the endpoint is the answer, not a copy of the question.