Frequently asked questions

Not to ask the questions. Someone technical deploys the server, registers the database connections and connects the AI client (docs/deployment.md §4, docs/datasources.md §3). After that a business question is a sentence, and a person who understands the data reviews the SQL and the numbers before release (docs/versioning.md §3).

Your database passwords stay on the server, encrypted (docs/datasources.md §7). A run reads each source in place and joins across sources in a temporary staging database dropped when the run ends (docs/staging.md §3); the result is kept briefly so your app or agent can page it (docs/rest-api.md §7). Rows and metadata the agent asks for do reach the agent, and so its model provider; a pipeline can also write results back to a database you choose (docs/pipeline-contract.md §8).

Today: pipelines authored with an AI agent, reviewed and released by a person, a published API for each released one, and datasets your Tableau workbook or your own application reads. Planned, without a date on this page: native dashboards, scheduled refresh, reports and alerts — the roadmap page carries the current order (docs/ROADMAP.md §2).

You look. Every step shows the rows it produced, the logic is readable SQL, and every run is recorded with its inputs so a number can be reproduced (docs/rest-api.md §10). Release checks you configure compare a run against expectations you declare (docs/pipeline-contract.md §12.12); they support the review, they do not replace it, and nothing goes live until you press Release.

Through your application: it supplies the customer's id as a declared parameter and holds the key. An endpoint key restricts which published paths it may call, not which rows a caller may see — row-level authorisation stays your backend's job (docs/rest-api.md §19.3, docs/auth.md §7.7). Per-viewer filtering for native dashboards is planned with the dashboards.

Today, yes for authoring: the surface is an agent over MCP or the REST API. The browser inspects a pipeline, executes it, and manages draft and released versions, but does not yet author one (docs/pipeline-editor.md §11). Browser authoring is a roadmap item (docs/ROADMAP.md §2).

Nothing to run: it is free and open source under AGPL-3.0 and you host it. There is no hosted plan today; when there is, the roadmap page will say so first. (Licence and packaging: docs/deployment.md §10.)

The eight engines are Postgres, Oracle, SQL Server, MySQL, H2, DuckDB, SQLite and dp-lake — dp-lake reads Parquet and Iceberg on S3 in place — and one dataset can read several of them in the same run (docs/datasources.md §4).

A self-hosted server that lets an AI coding agent — Claude Code, Cursor, Copilot, any MCP client — author SQL data pipelines across your databases, while people keep the release button; every released pipeline can be published as a versioned HTTP API. The tool surface is specified in docs/mcp-server.md §6.

A JSON document: named nodes, each a SQL template run against one datasource (or the per-run scratch engine), with explicit dependencies between them. The executor orders the graph, runs independent nodes in parallel, stages each result, and hands the last node's rows to the caller. The contract is docs/pipeline-contract.md §3 and the execution model docs/dag-executor.md §5.

Postgres, Oracle, SQL Server, MySQL, H2, DuckDB, SQLite and dp-lake — dp-lake is Parquet and Iceberg on S3 read in place by an embedded DuckDB. One pipeline can mix all of them; each node runs where its data lives. The dialect matrix with driver and licence notes is docs/datasources.md §4.

No. Every node runs at its source and streams its result into a per-run scratch engine (H2, created for the run and dropped after it), where the joining and shaping happen. The demo joins Postgres trips, MySQL weather, SQLite zones and Parquet rideshare data in one run. How staging works is docs/staging.md §3.

Yes — AGPL-3.0. Run it on your own infrastructure for your own company or your clients, read and change every line, keep your pipelines in your own database with no per-seat licence. The one obligation is the AGPL's: if you offer a modified server to others over a network, you publish the changes. The licence and the packaging are described in docs/deployment.md §10.

No. The agent holds an API key to datapipelines; the server holds the datasource credentials, encrypted at rest, and never returns them — not through a tool, not through the UI. A datasource can be marked read-only so write-shaped nodes are refused at save time. Credential storage is docs/datasources.md §7 and the read-only flag docs/datasources.md §3.

A single-database MCP server gives one agent one database and usually one connection string on a laptop. This gives the agent all of your databases through one scoped key, a scratch engine to join their results, versioned drafts a person reviews, an audit trail per call, and an HTTP API for what it builds. Every tool is specified in docs/mcp-server.md §6.2.

It can read schemas, catalogue statistics and indexes, probe a SELECT, create and iterate a draft pipeline, run it, read the result, and cancel its own runs. It can never release a version, never read a credential, and never touch a datasource marked read-only with a write. Scopes are per key and the matrix is docs/auth.md §7.6.

Yes. Every MCP tool call is written to the audit log with the key that made it and the tool — never the SQL text, row data or parameter values, which are your data; every lifecycle verb records who did it and through which surface (session, API key or MCP). The audit log is docs/auth.md §10 and the MCP events docs/mcp-server.md §10.

No, by design. Agents create and iterate drafts; a person releases. A released version is immutable — a later edit becomes a new draft — and a published endpoint serves the version you point it at. The lifecycle is docs/versioning.md §3.

A released pipeline gets a URL under /api/x/…; a GET runs it with the query string bound to its declared parameters and returns the rows as JSON. Keys are bound to a path prefix, so a partner can be given one endpoint and nothing else. The serving contract is docs/rest-api.md §19.

For the API half, today: an agent authors the pipeline against your databases, runs it, fixes what fails, you review and release, and the endpoint exists — no deployment, no service to write, no gateway to configure. Dashboards your agent creates and you embed in your own product are a planned roadmap item; the roadmap page says so. Publishing is docs/rest-api.md §19.

One Docker image beside a PostgreSQL metadata database and Redis. ./app.sh --start --demo nyc builds the image, downloads the published sample data and brings the stack up; the reference compose file and the demo quickstart are docs/deployment.md Appendix A and Appendix B.

Every statement carries a query timeout and the run an overall deadline; a node that outlives its budget fails with pipeline.node.query_timeout and the elapsed time — and the SQL probe returns the plan — so the agent fixes the shape instead of guessing. The limits and cancellation are docs/dag-executor.md §5.

Not today. It sits beside them: it can feed Tableau a governed dataset through an API or as Parquet on your own bucket, and it replaces the warehouse-plus-dbt step for teams that never wanted one. Dashboards, scheduling and alerts are on the roadmap page, undated. The comparison pages cite docs/versioning.md §3 for what is versioned.

Yes — all three families are real public data, not fixtures: NYC Taxi & Limousine Commission trip records and NOAA weather for New York, US Census trade and Federal Reserve exchange rates, and the TLC high-volume for-hire feed on S3. Every family is rebuilt deterministically from pinned sources, so the same version loads the same numbers anywhere. What ships is documented table by table on the demo-data page. (The build and its pins: docs/deployment.md Appendix B.)

That is what the licences are for. The NYC-derived families ship under NYC Open Data's no-restrictions statement; NOAA and the Federal Reserve are US Government works with no copyright; the Census slice carries the Bureau's API terms, including its not-endorsed notice; the Comtrade slice stays under the fee-free re-dissemination line. Each family's section on the demo-data page quotes the operative sentence and links the source. (The demo and its gates: docs/deployment.md Appendix B.)

The nyc family's files are about 118 MB and the trade family's about 223 MB, restored into the demo containers at start; the lake family is about 7.6 GB of Parquet and Iceberg objects that are never downloaded — the engine reads only the partitions a query names. The exact per-file and per-table figures render from the published manifests on the demo-data page, so they are always the published numbers. (The artifacts the loader fetches: docs/deployment.md Appendix B.)

Yes. The demo families are ordinary datasources — register yours in the app (or in the bootstrap file the demo profiles use) and any pipeline can join them to the demo tables, exactly as the shipped examples join Postgres trips to SQLite zones and MySQL weather. Nothing about the demo is privileged; it is seeded data with example pipelines. (Registration: docs/datasources.md §3.)

At start, yes: the nyc and trade artifacts download from the published bucket and the lake datasource points at S3. After that they differ — nyc and trade are local files restored into the demo engines, so running them needs no network, while the lake family is read in place over HTTPS at query time and always needs it. (The consuming side: docs/deployment.md Appendix B.)

Yes, two ways today. A released pipeline published as a GET endpoint returns JSON that Tableau's web data connector reads; or a pipeline writes its result to a datasource Tableau already connects to. Parquet on your own S3 bucket, read by Tableau natively, arrives with the lake write-back on the roadmap. The endpoint contract is docs/rest-api.md §19.

Not today, and the page does not claim it. Tableau draws; datapipelines gets the data there — across Postgres, MySQL, SQL Server, Oracle, SQLite and S3, without a warehouse, authored by an agent and released by a person. Embedded dashboards created by your agent are planned; the roadmap page carries the status. What is versioned today is docs/versioning.md §3.

Because the dashboard usually is not the expensive part — getting a trustworthy, joined, refreshed dataset into it is, and embedding the result in your own product is a second project. datapipelines makes the dataset a versioned pipeline an agent can author and an API you can call from your app today; the dashboards you embed are planned. Publishing is docs/rest-api.md §19.

For teams who would rather have their preparation as reviewable code than as a flow: a pipeline is a JSON document of SQL nodes with explicit dependencies, versioned, diffable, and runnable by an agent. It joins across engines in a scratch database the way Prep joins in memory, without a warehouse. The contract is docs/pipeline-contract.md §3.

Yes — free to run, for yourself or your customers, with every feature on every deployment. There is no seat count, no feature gate and no account with us, because the licence is AGPL-3.0 and you host it. (Licence and packaging: docs/deployment.md §10.)

Running it inside your company — even at scale, even commercially — asks nothing of you. If you modify the server and offer it to others over a network, you publish your modifications under the same licence; that is the whole obligation. (The licence: docs/deployment.md §10.)

There is no paid tier today, and nothing hosted is planned; if a hosted version ever exists it will be announced on the roadmap page first — never launched quietly. What is deliberately not planned lives on the roadmap's later band. (Source: docs/ROADMAP.md §3.)

Not today — there is nothing to buy. Help is GitHub Discussions and Issues, the docs under /docs, and the owner answers the contact address on the pricing page personally. You also own the ops: the upgrade runbook is docs/deployment.md §8.

The sample families are real public data, each under its own terms — NYC Open Data's no-restrictions statement, US Government public-domain works, the Census API terms — and the demo-data page quotes the operative sentence per family. (The demo and its gates: docs/deployment.md Appendix B.)

No. A catalog lists what exists — tables, columns, owners. This records what it means: a unit, a time zone, a grain, what a code means, how tables join, what revenue excludes — each with the query that proved it. The introspection surface it rides is docs/datasources.md §7.

That is what the trust ladder and the conflict rule are for: a fact is never higher-trust than its evidence, two live facts of the same kind are shown as a conflict — never a silent winner — and any fact can be retired with a reason. Facts are specified in docs/mcp-server.md §6.2.

No. The agent records what it learns while it works; you verify when you want to. Recording needs an author-scoped key and a release is still a person's act — the same bar as authoring a pipeline (docs/auth.md §7.6).

Drift is detected at read time: a renamed or dropped column flips its facts to stale, shown beside the current columns — nothing is silently re-mapped to a new column; someone re-verifies and records the successor. The drift rule is docs/metadata-db.md §4.18.

Yes — the skill and the tools are demo-free, and the facts come from whatever your agent probes: Postgres, Oracle, SQL Server, MySQL, H2, DuckDB, SQLite and dp-lake (docs/datasources.md §4).

Open a bug report on GitHub: https://github.com/msabiransari/datapipelines/issues/new?template=bug_report.yml. The form asks for the version (your deployment's /health build hash or the image tag), the surface it happened on, and the correlation id from the failed execution, the error toast or the REST envelope — so the report arrives carrying what a fix needs. What helps most during the beta is in CONTRIBUTING.md; the scope the beta covers is docs/ROADMAP.md §2.2.

GitHub Discussions: https://github.com/msabiransari/datapipelines/discussions. Questions about running it, ideas that are not yet feature requests, and show-and-tell all belong there — a concrete defect is a bug report and a formed proposal is a feature request, both one click away on the same page. What is already being built is the roadmap's known limitations, docs/ROADMAP.md §2.2, so check there first.

Privately, never as a public issue: open a private security advisory at https://github.com/msabiransari/datapipelines/security/advisories/new, or email the contact address in the footer if the advisory flow is not an option. What is in scope and what reporters are asked is SECURITY.md in the repository, and the security model a report is checked against is docs/auth.md.