Every data platform needs something to answer "run this, then that, retry if it fails, and tell me when it breaks." For a decade the answer was Apache Airflow, full stop. Now there are three serious choices — Airflow, Prefect, and Dagster — and they aren't three flavors of the same thing. They embody genuinely different philosophies about what you're even orchestrating: tasks, or the data assets those tasks produce. Get the philosophy right and the tool nearly picks itself. Get it wrong and you'll fight your orchestrator's worldview for years. This is a workload-first comparison, organized around the two axes that actually distinguish them.
The two axes: task-centric vs asset-centric (do you declare steps, or the data you want to exist?), and static vs dynamic (is the pipeline graph fixed at parse time, or built at runtime?). Airflow is task-centric and historically static; Prefect is task-centric and dynamic; Dagster is asset-centric. Those positions explain almost every difference that follows.
Apache Airflow: the incumbent, task-centric
Airflow is a workflow scheduler built around the DAG — a directed acyclic graph of tasks with dependencies, defined in Python, run by a scheduler against a metadata database. I covered how it works in Airflow Internals; the relevant fact here is its worldview: you orchestrate tasks. "Run extract, then transform, then load." Airflow's enormous advantage is gravity — it's everywhere, every cloud offers a managed version (MWAA, Cloud Composer, Astronomer), there's an operator/provider for virtually every system, and an army of engineers already knows it. Airflow 3.0 (2025) modernized it considerably — DAG versioning, a revamped React UI, a task-execution API that decouples workers, and stronger data-aware scheduling via assets — but the center of gravity is still scheduled task graphs.
Its honest weaknesses are the flip side of its history: it was built scheduler-first, not data-first, so "did this task succeed?" is native but "is this dataset fresh and correct?" was bolted on later. Dynamic, runtime-shaped pipelines fight the static DAG model, and top-level DAG-file code that runs on every scheduler parse is a classic foot-gun.
Prefect: Python-native and dynamic, task-centric
Prefect keeps the task-centric worldview but throws out the static graph. Pipelines are plain Python functions decorated as @flow and @task; the DAG is discovered by running the code, not by parsing a static definition. That makes dynamic workflows — loops, conditionals, fan-out whose width depends on runtime data — natural rather than awkward.
from prefect import flow, task
@task(retries=3)
def score(batch): ...
@flow
def nightly(batches):
for b in batches: # the graph's shape depends on runtime data — fine here
score.submit(b) # dynamic fan-out, no static DAG to predeclarePrefect's other defining choice is the hybrid execution model: Prefect Cloud (or a self-hosted server) handles orchestration and observability, but your code runs on your infrastructure — the control plane never needs to see your data. It feels the most like "just Python," which makes it a favorite for ML and data-science workflows where the pipeline is dynamic and code-heavy. The trade-offs: a smaller ecosystem of pre-built integrations than Airflow's, and — like Airflow — it orchestrates tasks, so data lineage and asset freshness aren't the native unit.
Dagster: asset-centric and data-aware
Dagster makes the genuinely different move. Instead of declaring tasks, you declare software-defined assets — the tables, files, and models you want to exist — and their dependencies on other assets. You don't say "run the transform task"; you say "this customers table is produced from these raw tables," and Dagster figures out execution. The orchestrator becomes data-aware by construction: it knows your assets, their lineage, their freshness, and their dependencies, because that's the unit you program in.
from dagster import asset
@asset
def raw_orders(): ...
@asset
def customer_ltv(raw_orders): # dependency is the ASSET, not a task ordering
return compute_ltv(raw_orders)
# Dagster knows the lineage raw_orders -> customer_ltv, its freshness, and how to rebuild itThat shift pays off in a built-in data catalog and lineage, freshness policies ("this asset should be no more than 2 hours stale"), a strong typing/testing story, and a development experience built around materializing and inspecting assets locally. The costs are real too: it's the most opinionated of the three, the asset mental model is a genuine learning curve for a team steeped in task DAGs, and its ecosystem, while growing fast (and strong on the dbt integration), is younger than Airflow's.
The core distinction, in one picture
graph TD
subgraph TASK["Task-centric (Airflow, Prefect)"]
T1["task: extract"] --> T2["task: transform"] --> T3["task: load"]
Q1["You declare: the STEPS to run.
Orchestrator tracks: did each task succeed?"]
end
subgraph ASSET["Asset-centric (Dagster)"]
A1["asset: raw_orders"] --> A2["asset: customer_ltv"] --> A3["asset: exec_dashboard"]
Q2["You declare: the DATA that should exist.
Orchestrator tracks: is each asset fresh & correct?"]
end
The worldview that drives everything else. Task-centric tools orchestrate steps and answer "did the job run?"; asset-centric Dagster orchestrates data products and answers "is this table fresh and correct, and what produced it?" Neither is wrong — but a team that thinks in datasets and lineage will fight a task scheduler, and a team that just needs jobs to run on time may find the asset model more ceremony than they need.
Side by side
| Airflow | Prefect | Dagster | |
|---|---|---|---|
| Core abstraction | Task DAG | Flow / task (Python) | Software-defined asset |
| Graph shape | Mostly static (3.0 improves) | Dynamic (runtime) | Asset graph (declarative) |
| Data awareness | Added later (assets) | Task-level | Native — lineage & freshness |
| Ecosystem / maturity | Largest, most battle-tested | Growing, Python-first | Growing fast, strong dbt story |
| Local dev / testing | Weakest of the three | Good (just Python) | Strongest — built for it |
| Managed options | MWAA, Cloud Composer, Astronomer | Prefect Cloud | Dagster+ |
| Sweet spot | Broad scheduled task graphs, existing skills | Dynamic, code-heavy / ML flows | Data-product platforms wanting lineage |
A decision guide
- Choose Airflow if you need the broadest ecosystem and battle-tested ubiquity, your team already knows it, or you want a managed offering on every cloud. It's the safe default for general scheduled orchestration, and 3.0 closed much of the modernization gap.
- Choose Prefect if your pipelines are dynamic and code-heavy — especially ML and data-science workflows — and you want orchestration that feels like plain Python, with a hybrid model that keeps your data on your infrastructure.
- Choose Dagster if you think in data assets and want lineage, freshness, and a strong testing/local-dev experience as first-class features — particularly if you're building a data-product platform and lean on dbt.
Don't rip out a working Airflow to chase a nicer model. Migrating orchestrators is a deceptively large project: it's not just rewriting DAGs, it's re-establishing every integration, every alert, every on-call runbook, and the team's hard-won operational intuition. The newer tools are genuinely better at the things they're better at — but "our Airflow works and everyone knows it" is a real, valuable asset that a slicker asset model rarely outweighs on its own. Switch when you have a concrete pain the new tool solves (you truly need asset lineage; your workflows are fundamentally dynamic), not because the demo looked cleaner. The orchestrator is load-bearing infrastructure; change it for a reason, not a vibe.
The choice is more reversible than it looks if you keep transformation logic out of the orchestrator. The teams that get badly locked in are the ones who put business logic inside operators and tasks. Keep your actual transformations in dbt, in well-factored Python packages, or in SQL the orchestrator merely invokes — and the orchestrator becomes a thin scheduling layer you could swap with far less pain. Whichever tool you pick, treat it as the conductor, not the orchestra.
What to carry away
Airflow, Prefect, and Dagster differ less in features than in worldview. Airflow and Prefect are task-centric — you orchestrate steps and the tool tracks whether they ran — with Airflow betting on ubiquity and ecosystem (and a real modernization in 3.0) and Prefect betting on dynamic, Python-native flows and a hybrid execution model. Dagster is asset-centric: you declare the data that should exist, and the orchestrator becomes data-aware, with lineage, freshness, and testing as native concepts.
So choose by how your team thinks. Need broad, proven scheduled orchestration with skills you already have? Airflow. Dynamic, code-heavy, ML-shaped workflows? Prefect. A data-product platform where lineage and freshness are the point? Dagster. And whatever you pick, keep your transformation logic out of the orchestrator so it stays a thin, swappable conductor — the orchestrator is one of the six undercurrents that runs beneath the whole lifecycle, not the place your business logic should live.