Built-in schedules, not an orchestrator · SQL pipelines across operational databases · agent-authored

Honest comparison

datapipelines.co vs Apache Airflow

The short version: if what you need is a scheduler that orchestrates work across your whole platform, use Airflow. It is mature, it is the standard, and we are not trying to replace it. We are a narrower thing, SQL pipelines across several operational databases, authored by an agent and run under governed credentials, and the two overlap less than the phrase "data pipeline" suggests. Read this as an alternative to Airflow only for that narrow job, never for the platform.

Read this first

Use Airflow instead when…

  • Orchestration is the problem. Dependencies between jobs, range backfills, retries with policies, SLAs, sensors that wait on external state. A schedule here runs one pipeline at a cron pattern's times in a time zone and records the run. It has none of the rest.
  • The work is not all SQL. Airflow orchestrates anything you can put in a Python operator: file transfers, ML training, API calls, container jobs. We execute SQL against JDBC datasources, plus JSONata or JavaScript SCRIPT nodes over the staged rows, and nothing else.
  • You need one control plane for the whole platform. A single place where every job's dependencies, history and alerting live. That is exactly what Airflow is for.
  • Your team writes Python and likes it. Airflow DAGs are Python. Our pipelines are declarative JSON referencing versioned templates, better for review and worse for arbitrary logic.

The other case

Use this instead when…

The data is in several operational databases

Reading Postgres, MySQL and SQLite in one query is the case this was built for. Each source is read in place. The join runs in a per-execution in-memory staging database, and nothing lands. In Airflow that is an extract job, a load job and a warehouse to keep.

An agent should be able to author, not just trigger

The full lifecycle is on the MCP surface: introspect schemas, create a template, create a pipeline, execute it, read the rows back. There is no equivalent shape where the authoring surface is a Python repository.

Giving an agent database access has to be safe

Datasources are read-only by default, with the refusal enforced at save time and again at execution against the live row. The documented pattern pairs the flag with a SELECT-only database user.

Every actor needs its own revocable credential

An Argon2id-hashed key per agent, an identity of its own holding the role its creator chose and pinned to a workspace, revocable in about a minute, with every tool call written to an audit log.

You want a small deployment

The app, a Postgres for metadata and a Redis for results. Schedules run inside the app: no separate scheduler service, no worker fleet, no executor to choose.

Reruns must be reproducible by construction

A released pipeline is immutable; editing one copies it into a draft. What ran last quarter is still there, byte for byte, without a git archaeology exercise.

They compose

These are not mutually exclusive. If you already run Airflow, an operator that POSTs to this server's execution endpoint and follows the SSE stream is a small task. Airflow keeps the schedule and the platform view, and the SQL that spans your operational databases stays here, versioned and governed. When the only need is one pipeline on a timetable, a built-in schedule is enough.

The dbt comparison → · Dagster vs Airflow, and where Prefect sits → · The cross-database join → · REST API specification →

Asked before

The Airflow comparison, in questions

Does this replace Airflow's scheduler?

Not as an orchestrator. A schedule here runs one SQL pipeline at a cron pattern's times in an IANA time zone (docs/scheduler.md §1), records every run and blocks when an outcome is unknown (docs/scheduler.md §5.2). It has no dependencies between schedules, no range backfills, no sensors and no task fan-out across other systems. If orchestrating many systems is the problem, Airflow is the answer.

Can Airflow trigger these pipelines?

Yes, and that is the composition the page recommends: an operator POSTs to the execution endpoint and follows the per-milestone SSE stream, so Airflow keeps the platform view while the cross-database SQL stays here. The execution API is docs/rest-api.md §6.

Can an agent author the work?

The full lifecycle is on the MCP tools: introspect schemas, create a template, create a pipeline, execute it and read the rows back. That is the shape an Airflow DAG repository does not offer a tool-calling agent. The tool list is docs/mcp-server.md §6.2.

Are reruns reproducible?

A released pipeline is immutable, and editing one copies it into a draft, so what ran last quarter is still exactly what ran, without a git archaeology exercise. The lifecycle is docs/versioning.md §3.1.

Keep Airflow. Give it something worth orchestrating.

The demo stack runs the cross-database pipeline an Airflow operator would trigger. See what the executed work looks like before you wire the two together.