Scope & vantage. Assessed mid-2026. Target use case: the distributed processing engine of a production data platform — batch transformation and enrichment, streaming where the business needs it, several teams sharing the deployment. The axes are engine-specific and are not comparable with my analytics-platform assessments. Ops Burden is inverted so that higher is better on every axis. Ray and Beam are legitimate omissions: Ray solves a different problem, and Beam is a portability layer that runs on two of these three.
1. Executive summary
Spark is the default and deserves to be; Flink wins outright when streaming is the point rather than a mode; and Arrow/DataFusion is quietly becoming the layer the other two are built on. That last one is why this assessment exists — the interesting story in 2026 is not Spark versus Flink, which has been stable for years, but the fact that a third thing has appeared underneath both.
The distinction that matters when choosing: Spark treats streaming as batch with small batches, and Flink treats batch as a stream that happens to end. Neither framing is wrong, and each is awkward at the other's home ground. Spark's micro-batch model gives you seconds, not milliseconds, and that is fine for the overwhelming majority of "real-time" requirements, which on inspection mean "faster than the nightly job". Flink gives you genuine event-at-a-time processing with exactly-once semantics that hold up, and charges you in developer experience and operational surface.
Arrow and DataFusion are a different category. You mostly do not deploy DataFusion as your platform's engine; you find it inside the tools you already run. It is scored here because "should we build on Arrow?" is now a real architectural question, and the answer is increasingly yes without it ever being a migration.
Verdict
Apache Spark — Adopt as the default batch engine. Unmatched ecosystem, the widest hiring pool of the three, and a decade of production scar tissue. Costs you a real operational surface and a developer loop that has improved but still lags.
Apache Flink — Adopt when streaming is the primary workload. Best-in-class latency and recovery semantics; the price is the steepest learning curve here and the heaviest ops.
Arrow / DataFusion — Assess. The best developer experience and lightest footprint on the page, and no distributed story to speak of. Build on it as a component; do not expect it to replace a cluster.
The risk nobody prices: engine choice is the least reversible decision on a data platform. Your transformation logic is written in its idiom, and that is where the migration cost lives — not in the infrastructure.
2. Positioning
These three disagree about what a data pipeline fundamentally is, and every other difference follows from that.
Spark says a pipeline is a job: read a bounded dataset, transform it, write it. Structured Streaming grafted continuous processing onto that model by making the batches very small, which is an elegant reuse of a mature engine and also the source of its latency floor. Flink says a pipeline is a continuous dataflow with state, and a batch job is just one whose input eventually stops. That inversion is why Flink's checkpointing and exactly-once guarantees are stronger, and why its batch story has always felt like the afterthought it is. DataFusion says a pipeline is a query plan, and that the interesting work is making one process columnar memory as fast as physics allows on a single machine.
| Dimension | Apache Spark | Apache Flink | Arrow / DataFusion |
|---|---|---|---|
| Core model | Batch; streaming as micro-batch | Stream; batch as bounded stream | Vectorised query plan over Arrow memory |
| Latency floor | Seconds | Milliseconds | In-process |
| State handling | Improving, bolted on | First-class, RocksDB-backed | Not its problem |
| Written in | Scala / JVM | Java / JVM | Rust |
| Distribution | Mature, everywhere | Mature | Single-node; Ballista is early |
| You meet it as | A platform you deploy | A platform you deploy | A library inside other tools |
| Fails when | Nobody owns partition sizing and shuffle | Nobody understands checkpointing | You expected a cluster |
The 2026 development worth naming: Arrow has won the memory format argument so completely that it is now infrastructure rather than a choice. Spark reads it, Polars is built on it, DuckDB speaks it, and the native execution engines I covered in Comet, Gluten and Photon are all bets on vectorised columnar execution over exactly this substrate. Choosing "Arrow" increasingly means noticing you already did.
3. Radar criteria — the reasoning
Batch throughput
Spark 4.8 — this is its home field and a decade of optimiser work shows. Arrow/DataFusion 4.2: extraordinary per-core efficiency, limited by having no serious distributed execution. Flink 4.0, genuinely capable at batch and clearly not what the engine was designed around.
Streaming & latency
Flink 4.8 and it is not close. True event-at-a-time processing, watermarks that handle late data properly, and exactly-once that survives contact with a failure. Spark 3.9: Structured Streaming is good, well-integrated and floor-limited by micro-batch — see my real-time mode piece for how far that floor has moved. DataFusion 3.0, essentially not a streaming engine.
Developer experience
DataFusion 4.2 — a library you call, with no cluster to reason about, which is a category advantage rather than a fair fight. Spark 3.8, helped enormously by the DataFrame API and dampened by the gap between what you write and what executes. Flink 3.4: the most concepts to hold in your head before the first job runs.
Language support
Spark 4.6 — Scala, Python, Java, R and SQL, all genuinely first-class, and PySpark is why most teams can staff it. Flink 3.9: Java-first, with PyFlink real but second-class. DataFusion 3.6, Rust-native with Python bindings.
Scale & elasticity
Spark 4.7, Flink 4.5 — both scale to thousands of cores routinely, Spark with more people who have already done it at your size. DataFusion 3.2: single-node by design; Ballista exists and is not something I would put in front of a client yet.
Reliability & recovery
Flink 4.6 — distributed snapshots are the strongest recovery model in this class, and the reason Flink survives in payments and fraud. Spark 4.4, very solid; checkpoint-and-restart is coarser but easier to reason about at 3am. DataFusion 3.4, which is really the host application's problem.
Observability
Spark 4.0 — the Spark UI is genuinely good once you know which four pages matter. Flink 3.8: excellent metrics, and diagnosing backpressure remains a specialist skill. DataFusion 3.2, EXPLAIN and whatever your host exposes.
Ecosystem & connectors
Spark 4.9, the highest score in this assessment and the axis that keeps winning arguments — a connector exists for everything, and when it doesn't someone has written the workaround. Flink 4.1, strong on the streaming side. DataFusion 3.8, growing fast from a small base.
Light ops (inverted — higher means less to run)
DataFusion 4.6 — a dependency, not a deployment. Spark 3.4: cluster sizing, shuffle tuning, and the memory settings nobody wants to own. Flink 3.0, the heaviest here — checkpoint storage, state backends, and savepoint discipline on every upgrade. Managed offerings move both JVM engines up roughly a point.
TCO / cost control
DataFusion 4.6 — no cluster, no bill. Spark 3.6 and Flink 3.5: both dominated by the compute they schedule rather than the software, and both easy to overspend on when nobody owns right-sizing. As with orchestration, this axis should rarely decide the choice.
4. Radar scorecard
How these scores are arrived at — the rings, the 0–5 scale, and the evidence rules — is written up in the radar methodology. This table and the chart below are generated from one file, scorecards.json: the source of truth, machine-readable, and kept separate so two copies of the same numbers cannot drift apart between editions. Last reviewed 2026-Q3.
| Criterion | Apache Spark | Evidence | Risk → mitigation |
|---|---|---|---|
| Batch Throughput | 4.8 | A decade of optimiser work; the batch default | Micro-batch latency floor → Flink when milliseconds matter |
| Streaming & Latency | 3.9 | Structured Streaming, well integrated | Seconds not milliseconds → check the actual SLA, not the word 'real-time' |
| Developer Experience | 3.8 | DataFrame API; large improvement over RDDs | Plan/execution gap → read the physical plan, not the code |
| Language Support | 4.6 | Scala, Python, Java, R, SQL all first-class | — |
| Scale & Elasticity | 4.7 | Thousands of cores, routinely, everywhere | Shuffle and skew → partition sizing needs an owner |
| Reliability & Recovery | 4.4 | Checkpoint-and-restart; coarse but legible | — |
| Observability | 4.0 | Spark UI; good once you know the four useful pages | — |
| Ecosystem & Connectors | 4.9 | A connector for effectively everything | — |
| Ops Burden (inverted: higher = lighter) | 3.4 | Cluster sizing, shuffle and memory tuning are yours | Ops load → managed Spark, or a funded platform owner |
| TCO / Cost Control | 3.6 | Dominated by scheduled compute, not licence | Overspend → autoscaling + right-sizing with a named owner |
Comparator scores (same axis order) — Apache Flink: 4 / 4.8 / 3.4 / 3.9 / 4.5 / 4.6 / 3.8 / 4.1 / 3 / 3.5. Arrow / DataFusion: 4.2 / 3 / 4.2 / 3.6 / 3.2 / 3.4 / 3.2 / 3.8 / 4.6 / 4.6.
5. The radar
Ten engine axes, Ops inverted so higher is always better. The shapes tell the story: Spark is broad and dips at ops, Flink spikes hard at streaming and recovery while paying for it everywhere operational, and Arrow/DataFusion is the inverse of both — superb where there is no cluster, absent where there is one. Click any name in the legend to toggle it on or off — comparing two shapes at a time is far more legible than comparing four.
6. Workload-specific analysis
| Workload | Best fit | Notes |
|---|---|---|
| Nightly batch transformation at scale | Spark | The default, and the boring correct answer. |
| Sub-second event processing | Flink | Fraud, payments, alerting. The only one of the three that genuinely does this. |
| Exactly-once with complex state | Flink | Distributed snapshots are the strongest model here. |
| ML feature pipelines | Spark | Ecosystem and Python support settle it. |
| Embedded / single-node analytics | DataFusion | Or DuckDB, which is the same instinct with more polish. |
| Building a query engine of your own | DataFusion | The reason it exists. Don't rewrite a vectorised executor. |
| Data under ~100 GB | None of these | You do not need a cluster. Polars or DuckDB on one machine will beat all three. |
The trap: choosing a distributed engine for data that fits on a laptop. Cluster-scale tooling has a fixed cognitive and operational cost that does not shrink with your data, and a surprising share of Spark deployments I have reviewed process volumes a single machine would handle in seconds. The instinct is understandable — nobody wants to re-platform later — but you pay the ops tax every day and the re-platform is a week. Measure the actual volume before choosing the engine, and be honest about whether "we'll grow into it" has a date attached.
7. Reference architecture
The pattern I would build: Flink where latency is the requirement, Spark where volume is, and the same open table format underneath both so the choice stays reversible at the storage layer even though it is expensive at the logic layer.
graph LR K["Kafka"] --> F["Flink
low-latency enrichment"] S["Object storage
batch drops"] --> SP["Spark
bulk transformation"] F --> T[("Open table format
Iceberg / Delta")] SP --> T T --> Q["Query engines"] Q --> BI["BI"] Q --> ML["ML training"] T -.->|"read in place"| DF["DataFusion / DuckDB
embedded analytics"] CP[("Checkpoint store")] -.->|"state + recovery"| F ORC["Orchestrator"] -.->|"triggers"| SP
Two engines, one storage layer. The dotted paths are the ones that decide whether this survives: Flink's checkpoint store is the difference between a recoverable job and a rebuild, and the orchestrator triggers Spark rather than Spark scheduling itself.
The Flink configuration that separates a job that recovers from one that silently loses state:
// Exactly-once is a property of the whole pipeline, not a checkbox.
// The sink has to participate, or you have exactly-once processing and
// at-least-once delivery — which is at-least-once.
env.enableCheckpointing(30_000, CheckpointingMode.EXACTLY_ONCE);
CheckpointConfig cfg = env.getCheckpointConfig();
cfg.setMinPauseBetweenCheckpoints(10_000); // don't checkpoint back-to-back
cfg.setCheckpointTimeout(120_000);
cfg.setTolerableCheckpointFailureNumber(3); // fail the job, don't drift blind
// Keep the state on an upgrade: without this, a cancel discards it.
cfg.setExternalizedCheckpointCleanup(
ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
// RocksDB once state outgrows heap. Incremental, or every checkpoint
// rewrites the whole thing and the job stalls behind its own snapshots.
env.setStateBackend(new EmbeddedRocksDBStateBackend(true));8. POC plan (4 weeks)
- Week 1 — measure the data, not the ambition. Actual volume, actual arrival pattern, actual freshness requirement in seconds. If the answer is 40 GB and "within the hour", stop the POC and use a single machine.
- Week 2 — port the ugliest job. The one with the skewed join or the awkward window. Hello-world runs beautifully on all three and tells you nothing.
- Week 3 — kill it mid-run. Fail a task, restart from checkpoint, replay a day, and change a schema underneath a running job. Recovery ergonomics are what you will actually live with.
- Week 4 — cost the operating model. Who owns cluster sizing? Who is paged when a checkpoint times out? Re-score with your numbers and write the ADR naming the deciding axis.
9. Final recommendation
Adopt Spark as the default. Adopt Flink where streaming is the product. Assess Arrow/DataFusion as a component, not a platform.
Spark remains the right default and the reasons are unglamorous: the ecosystem is unmatched, the hiring pool is an order of magnitude deeper than Flink's, and the failure modes are documented by thousands of people who hit them first. For batch transformation at scale it is the boring correct answer, and boring correct answers are what platforms are made of.
Flink earns its place when the requirement is genuinely sub-second and stateful — fraud scoring, payment authorisation, real-time alerting. Its checkpointing model is the strongest recovery story in this class. But be honest about the requirement first: most "real-time" specifications turn out to mean minutes, and Spark serves those without the operational surface Flink demands.
Arrow and DataFusion I would design toward rather than deploy. The memory format has effectively won, and building on it is increasingly a decision you make by default. DataFusion as your platform's distributed engine is not a 2026 recommendation; DataFusion inside a service you are writing very much is.
The thing to internalise: this is the least reversible decision on the page. Infrastructure is replaceable, but every transformation you write is written in an engine's idiom, and that is what a migration actually costs. Choose as though you cannot change your mind for five years — and re-assess when Ballista or an equivalent gives DataFusion a credible distributed story, because that is the change that would redraw this.
References
Two of these three have peer-reviewed papers describing their design, which is unusually good evidence for this category.
Research
- Zaharia et al., "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing", NSDI 2012. The RDD paper — still the clearest explanation of why Spark recovers the way it does.
- Carbone et al., "Lightweight Asynchronous Snapshots for Distributed Dataflows", arXiv 2015. Flink's checkpointing algorithm, and the reason its exactly-once claim is more than marketing.
- Armbrust et al., "Spark SQL: Relational Data Processing in Spark", SIGMOD 2015. Catalyst, and how the DataFrame API became the thing people actually use.
Primary sources
- Spark — Structured Streaming guide. The micro-batch model, stated by its authors.
- Flink — stateful stream processing. State, checkpoints and the guarantees that follow.
- Apache DataFusion and the Arrow columnar format spec.
Deeper reading (blog)
The internals behind the scores:
- Apache Flink Internals — checkpoints, watermarks, and what exactly-once costs.
- Spark Native Engines: Comet, Gluten, Photon — the vectorised execution bet, all built on Arrow.
- Spark Real-Time Mode — how far the micro-batch floor has actually moved.
- Arrow & DataFusion Internals — the columnar standard underneath everything.
- pandas vs Polars vs DuckDB vs Arrow — the single-machine answer most teams actually need.
- Radar: Airflow vs Dagster vs Prefect — what triggers all of this.
Scores are my own architect-level judgment for the stated use case, calibrated to mid-2026 — not a vendor ranking. Processing-engine axes; NOT comparable with the analytics-platform assessments. Re-score against your own POC numbers before you commit.