Just use Postgres — until one of these arrives
The honest answer to the question in the search bar: probably not yet. If one database holds the data and the people asking can query it, Postgres alone is the right architecture, and adding a warehouse would be adding a system to run for nothing. This page is about the four arrivals that change the answer — the second database, the agent that needs read-only access, the partner who wants an API, and the release step nobody wrote — and what each one actually needs, which is less than a warehouse.
Said plainly
One database, one team, questions the database itself can answer: that is Postgres doing its job, and it does it superbly. Materialized views, read replicas and a good schema cover most of what a first dashboard needs. A warehouse earns its keep when data is spread across systems that cannot see each other, or when the volume forces a columnar engine — and a team that is not there yet should not pretend otherwise. Buying the warehouse early buys a second deployment, a second security perimeter and a sync to keep honest, in exchange for nothing the database had not already answered.
This page does not try to sell you out of that position. datapipelines does not replace Postgres here — there is nothing for it to do yet. What the page maps is the boundary: the four arrivals that make "just use Postgres" stop being the whole answer, and the smallest thing that meets each one.
How do you know you are close to the boundary? The tells are cheap to watch for. The spreadsheet that joins an export to a lookup table by hand, weekly. The second database that appeared for a feature and now holds data someone asks about. The script someone wrote to "just copy one table" between systems, which nobody owns and everybody runs. None of these is a warehouse-sized problem — that is exactly the point. They are join-shaped, access-shaped and version-shaped problems, and reaching for a warehouse to solve them is how a small team ends up running a platform.
The four arrivals
The question crosses engines now — orders in Postgres, events in MySQL, a lookup in SQLite — and no engine can see two. What is needed is a place for the join: a scratch database created for that one run, sources staged into it, dropped after. Not a warehouse; a per-run workbench.
An AI coding agent is the newest member of the team and the one you least want holding a superuser password. The server-side shape: credentials encrypted at rest on the server, the agent holding a scoped, revocable key, read-only flags where writes are out of bounds — and every call audited.
A partner, a customer or another team asks for their slice of the data on a URL. Handing out database credentials is not that answer. A released pipeline published as a GET endpoint — parameters bound, keys scoped to its path — is, and it is one release away, not a service to write.
Once more than one person can change a query, the question "which version produced this number?" needs an answer that is not archaeology. Draft, review, release — by a person, immutable once released — is the missing control, and it is structural here rather than a convention in a README.
What this adds
Postgres stays exactly where it is, serving your application. It becomes a datasource: a named, governed connection the agent reads through and a pipeline can query — marked read-only wherever nothing should ever write. The agent's SQL runs against your real schema with a key you issued and can revoke; the database password itself never leaves the deployment.
The scratch engine is per run, so there is nothing new to operate when the second source arrives — the join happens in an H2 workbench that exists for exactly that execution. And when the result of a pipeline belongs back in Postgres — a rollup table a dashboard reads, say — a write-back node lands it in a database you already run. You only meet a second deployment if you decide the data volume, not the question count, has earned one.
The order of operations matters, and it is the order this page is written in: the datasources register in minutes, the agent's first draft is a review artefact rather than a commitment, and the release step only appears when a second person cares what ran. Postgres was never demoted; it gained a governed front door for the questions that stopped fitting inside a single engine. If and when a warehouse genuinely arrives — volume, retention, a modelling practice — the pipelines you built in the meantime speak the same endpoint contract into it, which is a migration path, not a cliff. And nothing here is a rewrite of how your team works: the database stays the database, and the new arrivals plug into it rather than around it.
The decision, short
What it looks like
| datasource | what it is | access |
|---|---|---|
| sample-lake | Parquet on S3, read in place | read-only ✓ |
| sample-reference | SQLite | read-only ✓ |
from the demo workspace — the agent sees the schema, never the credential
The demo is the small case made real: ./app.sh --start --demo nyc seeds Postgres trips,
MySQL weather and SQLite zones, and nyc/mobility/revenue_by_borough joins two of them in one
pipeline — the first arrival, answered without a warehouse in sight.
Then register your own Postgres beside the demo sources and ask the agent the question your team answered
by hand last week. If the honest verdict is "Postgres alone still covers this", the page has done its
job: you have tested the boundary instead of assuming it, and the demo stack uninstalls with
./app.sh --clean.
Asked before
When one database holds the data and the people asking questions can query it directly: no second system to keep, no copy to go stale, and a warehouse would be an empty room. The moment a second engine arrives, a cross-engine join needs somewhere to happen — docs/staging.md §3 is where it happens here.
Four things: a question that crosses engines, which needs a per-run scratch database to join in; credential sprawl, which stops when the server holds the credentials encrypted and the agent holds only a key; an API someone will ask for; and a version a person released behind it. Credential storage is docs/datasources.md §7.
No. Postgres becomes a datasource: it keeps serving your application, the agent reads it through the server with a scoped key, and a pipeline can write its result back into a Postgres table if that is where the answer belongs. Write-back is docs/pipeline-contract.md §8.
A person. The agent authors and iterates a draft and can run it, but releasing — the step that makes the pipeline what production serves — is a human verb, and a released version is immutable. The lifecycle is docs/versioning.md §3.
Bring up the stack, register your own Postgres alongside the demo sources, and watch the agent draft the first cross-source pipeline. The warehouse decision can wait until the data, not the workflow, demands it.