Diff in a pull request
The pipeline is JSON referencing named SQL templates; a change is a line-level diff a reviewer reads before it merges. A flow file changes wholesale and invisibly.
Same job, different artefact
Tableau Prep and a datapipelines pipeline do the same first job — pull several sources together, shape them, hand the result to the next step. The difference is the artefact that job produces. Prep produces a flow: a document one tool opens, edits and runs. A pipeline here is a JSON document of SQL nodes: diffable in a pull request, versioned, runnable by an AI agent, releasable only by a person, and publishable as an API. This page compares them honestly, including the part Prep does better.
The artefact
A Prep flow is a diagram you edit. A pipeline here is a JSON document of SQL nodes with explicit dependencies, which happens to render as a diagram. The distinction sounds academic until the second week, when the questions change from "can I build it?" to "can I review it, diff it, hand it to someone else, and say which version of it produced the number on the board?". The flow answers those in person and from memory. The pipeline answers them structurally: the diff is a pull request, the review is a draft a person can run before releasing, and the version is immutable once released.
Both join across sources without a warehouse. Prep joins in memory on the machine running the flow; a pipeline stages each source's rows into a scratch database created for the run and joins them there, with ordinary SQL any reviewer can read. When a source is slow, the difference gets loud: the pipeline's node carries a query timeout, and the SQL probe returns the plan, so the agent fixes the shape from evidence rather than from a spinner.
There is a quieter difference underneath, and it decides who can help you. The flow's knowledge lives in the person who built it and in the file only that tool opens; hand the work to a colleague, a contractor or an agent, and the first week is archaeology. The pipeline's knowledge lives in the document: the nodes are named for what they do, the SQL is in the file, the dependencies are declared — so the second person (or the second agent) starts from reading, not from reverse-engineering. Teams do not adopt pipelines because the first build is faster; often it is not. They adopt them because the second week, the second person and the second year are all cheaper.
The list
The pipeline is JSON referencing named SQL templates; a change is a line-level diff a reviewer reads before it merges. A flow file changes wholesale and invisibly.
The agent's work lands as a draft. A person runs it, reads the rows, and releases it or not. The released version is immutable; the next edit becomes a new draft.
The whole authoring surface is MCP tools: schemas in, catalogue statistics and indexes read, the draft created and iterated, the run inspected. A flow has no interface an agent can work through.
Every statement carries a query timeout, and the SQL probe captures the plan before the query runs — so a slow node names itself, with a plan, instead of being a spinner in someone else's UI.
A node can land its result in a datasource — the Postgres table a BI tool live-connects to. The flow's output stays where the flow puts it.
Release the pipeline, publish the endpoint: a versioned GET with bound parameters and a key scoped to its path. That is the step a flow has no answer for.
The honest column
Visual profiling — the distributions, the nulls, the value counts a flow shows while you drag — is a real strength, and this server does not ship an equivalent: the agent reads catalogue statistics and the probe's rows instead, which is evidence, not a histogram. The click-to-pivot is faster than writing the SQL by hand if you do not have an agent to write it. And Tableau-native output — the flow that lands directly where a Tableau author expects — is friction this does not attempt to match; the honest path here is the write-back node or the endpoint.
If the preparation is a one-off reshape by a person who lives in Prep, none of the six above pays for the move. The pipeline earns its keep when the preparation is repeated, reviewed, handed between people and machines, or served — which is the moment a file one tool opens stops being enough.
The move
Pick the flow you would least like to rebuild from scratch next quarter — the weekly one, with the joins someone has memorised. That flow pays for the migration twice: once in the move, and once every week it no longer depends on the person who built it. The five steps below are each a shipped feature; the whole sequence is an afternoon for a flow of ordinary size.
List every input the flow touches — databases, files, the join keys between them. This list is the migration's scope; most flows are smaller than they look once the inputs are named.
Each source becomes a datasource: name, dialect, connection, credentials the server encrypts and never returns. Mark read-only every source the pipeline must never write to — the refusal is enforced at save time and again at run time.
The flow's steps are a spec: describe the shape in business terms. The agent reads the real schemas and catalogue statistics, drafts one node per source plus a join in the scratch database, runs it, and reads the failure when there is one.
Run the flow once more and the pipeline once, and compare row counts and spot totals. The pipeline's result is paged rows you can read in the results dock — the comparison is a morning, not a project.
When the numbers match, a person releases. The flow on the shared drive can retire; the pipeline is versioned, diffable, and — the step Prep never had — publishable as an endpoint.
What it looks like
you › Which rideshare company carried the most trips in each borough last quarter?
| borough | top company | trips | borough trips | share |
|---|---|---|---|---|
| Manhattan | Uber | 17,660,839 | 23,669,163 | 74.62% |
| Brooklyn | Uber | 11,946,362 | 16,057,699 | 74.40% |
| Queens | Uber | 9,786,602 | 12,847,231 | 76.18% |
| Bronx | Uber | 6,078,750 | 7,572,899 | 80.27% |
| Staten Island | Uber | 715,593 | 918,360 | 77.92% |
GET /api/x/demo/top-company-by-borough?anchor_date=2025-01-01 · after release · key scoped to /demo
from the demo workspace — a real result, not a promise
Both panels are the demo workspace's run of record — demo/top_company_by_borough, a
recorded acceptance result over the published sample data (the reference zones and the lake's
ride ledger on S3), reproduced from the bucket. The same sentence, the same steps and the same
numbers the home page shows.
Asked before
Yes: a pipeline is a JSON document of named SQL nodes with explicit dependencies, so it diffs in a pull request, carries a version number, and is reviewed the way code is. A flow file is a diagram only one tool opens. The contract is docs/pipeline-contract.md §3.
Visual profiling and the click-to-pivot are real strengths, and a one-off reshape by a person who lives in Prep is not a problem this needs to solve. The pipeline earns its keep when the preparation must be repeatable, reviewable and agent-authored — the authoring surface is docs/mcp-server.md §6.2.
That is the intended workflow: the agent reads your schemas, catalogue statistics and indexes, drafts the pipeline, runs it, and reads the failure when there is one — then leaves a draft that a person releases. The tool surface is docs/mcp-server.md §6.2.
In five steps, each a shipped feature: inventory the flow's sources, register them as datasources (read-only where nothing should write), describe the transformation to your agent, run the pipeline and compare row counts with the flow's output, then release. Registering datasources is docs/datasources.md §3.
The demo stack gives you the four sources to practise on. Migrate the smallest flow you run weekly — the comparison against its output is the whole argument, whichever way it lands.