datapipelines.co vs Apache Airflow

The short version: if what you need is a scheduler that orchestrates work across your whole platform, use Airflow. It is mature, it is the standard, and we are not trying to replace it. We are a narrower thing — SQL pipelines across several operational databases, authored by an agent and run under governed credentials — and the two overlap less than the phrase "data pipeline" suggests.

Read this first

Use Airflow instead when…

Scheduling is the problem. Cron-style triggers, backfills, retries with policies, SLAs, sensors that wait on external state. We have none of that: pipelines here are executed on demand, by an agent, a REST call or a person.
The work is not all SQL. Airflow orchestrates anything you can put in a Python operator — file transfers, ML training, API calls, container jobs. We execute SQL against JDBC datasources, and nothing else.
You need one control plane for the whole platform. A single place where every job's dependencies, history and alerting live. That is exactly what Airflow is for.
Your team writes Python and likes it. Airflow DAGs are Python. Our pipelines are declarative JSON referencing SQL templates — better for review, worse for arbitrary logic.

The other case

Use this instead when…

The data is in several operational databases

Reading Postgres, MySQL and SQLite in one query — each in place, joined in a per-execution in-memory staging database, nothing landed — is the case this was built for. In Airflow that is an extract job, a load job and a warehouse to keep.

An agent should be able to author, not just trigger

The full lifecycle is on the MCP surface: introspect schemas, create a template, create a pipeline, execute it, read the rows back. There is no equivalent shape where the authoring surface is a Python repository.

Giving an agent database access has to be safe

Datasources are read-only by default, with the refusal enforced at save time and again at execution against the live row. The documented pattern pairs the flag with a SELECT-only database user.

Every actor needs its own revocable credential

One Argon2id-hashed key per agent, scoped and pinned to a workspace, revocable in about a minute, with every tool call written to an audit log.

You want a small deployment

The app, a Postgres for metadata and a Redis for results. No scheduler, no worker fleet, no executor to choose.

Reruns must be reproducible by construction

A released pipeline is immutable; editing one copies it into a draft. What ran last quarter is still there, byte for byte, without a git archaeology exercise.

They compose

These are not mutually exclusive. If you already run Airflow, an operator that POSTs to this server's execution endpoint and follows the SSE stream is a small task — Airflow keeps the schedule and the platform view, and the SQL that spans your operational databases stays here, versioned and governed.

The dbt comparison → · The cross-database join → · REST API specification →

Asked before

The Airflow comparison, in questions

Does this replace Airflow's scheduler?

No — pipelines here execute on demand, by an agent, a REST call or a person; there are no cron triggers, backfills or SLAs. If scheduling is the problem, Airflow is the answer, and the page says so first. The execution model is docs/dag-executor.md §5.

Can Airflow trigger these pipelines?

Yes, and that is the composition the page recommends: an operator POSTs to the execution endpoint and follows the per-milestone SSE stream, so Airflow keeps the platform view while the cross-database SQL stays here. The execution API is docs/rest-api.md §6.

Can an agent author the work?

The full lifecycle is on the MCP surface — introspect schemas, create a template, create a pipeline, execute it, read the rows back — which is the shape an Airflow DAG repository does not offer a tool-calling agent. The tool list is docs/mcp-server.md §6.1.

Are reruns reproducible?

A released pipeline is immutable — editing one copies it into a draft — so what ran last quarter is still exactly what ran, without a git archaeology exercise. The lifecycle is docs/versioning.md §3.1.

Keep Airflow. Give it something worth orchestrating.

The demo stack runs the cross-database pipeline an Airflow operator would trigger — see what the executed work looks like before you wire the two together.

Run the demo How results become endpoints