Two orchestrators · one honest decision · no product pitch until the end
Editorial comparison
Both are open-source, Python-first orchestrators under the Apache-2.0 licence, and both will run a nightly job that moves data from A to B. The difference is what each asks you to describe: Airflow asks for the work — a graph of tasks and when to run it — and Dagster asks for the data — the tables and files your code produces, and what each depends on. That one difference decides most of the rest. Prefect asks for neither: it takes the Python functions you already have and makes them observable.
In their own words
Airflow calls itself "an open-source platform for developing, scheduling, and monitoring workflows", batch-oriented data pipelines and ML training included. A DAG carries the schedule, the tasks, their dependencies and callbacks; workflows are "defined entirely in Python"; backfills and re-running failed tasks are built in. Running it means a scheduler, a DAG processor, an API server, a metadata database and, usually, workers.
Dagster calls itself "a data orchestrator built for data engineers, with integrated lineage, observability, a declarative programming model, and best-in-class testability". Its unit is the asset — "an object in persistent storage, such as a table, file, or persisted machine learning model" — and materializing one means running its function and saving the result. Asset definitions know their own dependencies, so lineage comes free; schedules, sensors and automation conditions decide when they run.
Prefect calls itself "an open-source orchestration engine that turns your Python functions into production-grade data pipelines with minimal friction". Flows "are defined as decorated Python functions" and can use native control flow — conditionals, loops, tasks created at runtime — rather than a graph declared up front. The server is self-hosted or Prefect Cloud.
The fork
Ask one question first: when something breaks at 3 a.m., do you want to be told which task failed, or which table is now stale and who reads it? Teams that answer "task" are describing Airflow; teams that answer "table" are describing Dagster. Everything below is that question, applied.
The third name in the search box
Prefect is the answer for a team whose pipelines are already Python functions and who want them scheduled,
retried and observed without rewriting them as a graph. Where Airflow wants the DAG declared up front, a
Prefect flow is ordinary Python — an if, a loop, tasks created from the data at hand — with the
orchestration added by a decorator. That makes it the lightest of the three to adopt and the least
opinionated about what your data looks like afterwards.
A search phrase, answered
The honest candidates are the two above. Airflow if you want the standard and its ecosystem; Prefect if you want the least ceremony around Python you already have. There are other orchestrators, and each has a case, but a team leaving Dagster is almost always choosing between "more platform" (Airflow) and "less model" (Prefect). What no orchestrator is an alternative for is the SQL itself — the joins and the reviewed datasets the tasks produce — which is the one place this site's own product enters, below.
Whichever you choose, it will orchestrate work. When the work is SQL across several operational databases, authored by an agent and run under governed, read-only-by-default credentials, that part can live here: a versioned pipeline the orchestrator triggers through one endpoint and follows to completion. datapipelines has no scheduler and wants none — it composes with Dagster, Airflow or Prefect rather than competing with any of them. The Airflow comparison spells out the boundary; the dbt comparison covers the modelling side of the same question.
datapipelines vs Airflow → · datapipelines vs dbt → · REST API specification →
Asked before
No. Pipelines execute on demand — by an agent over MCP, by a REST call, or by a person in the editor — and there is no scheduler inside this server: no cron triggers, no backfills, no sensors. If the work needs those, that is the orchestrator's job, and the page says which one fits which team. The execution lifecycle is docs/dag-executor.md §5.
Yes, and that is the composition the page recommends: a task in any of the three POSTs to the execution endpoint and follows the per-milestone SSE stream until the run reaches a terminal state, so the orchestrator keeps the platform view while the SQL that spans your operational databases stays here, versioned and governed. The endpoint is docs/rest-api.md §6.
Not in the sense a searcher means. Those two orchestrate whole platforms of Python work; this server runs SQL pipelines across several operational databases, authored by an agent and executed under governed, read-only-by-default credentials — a narrower thing that composes with either rather than replacing it. What it does run is docs/dag-executor.md §5; what an agent can author is docs/mcp-server.md §6.1.
From each project's own current documentation, cited by URL in the page source next to every claim — nothing about them here is written from memory, and the version each site showed on the day it was read is named in the citation. The claims about this server rest on its own specs the same way: the execution model is docs/dag-executor.md §5.
The demo stack runs the cross-database pipeline any of the three would call — read the SQL, the run and the result before you wire anything together.