Every time I've tried to make Apache Iceberg behave like a real-time table, I've ended up fighting the same fight: a Flink job upserting a CDC stream, and the table slowly drowning in tiny files and merge-on-read overhead until queries crawl and I'm babysitting a compaction schedule like it's 2013 and I'm running HBase again. Iceberg is a superb format. It just wasn't born for a firehose of row-level updates — it was born for large, mostly-append analytical tables, and streaming upserts got added to a design that started somewhere else. You can feel the seam.
Apache Paimon is what you get when someone starts from the opposite end. Instead of a batch table format that learned to stream, it's a streaming table format — an LSM-tree, the same structure inside RocksDB and ClickHouse, living directly on object storage — that also does batch. It graduated to an Apache Top-Level Project earlier in 2025, having grown out of what used to be Flink Table Store, and it fills a genuine gap the Iceberg/Hudi/Delta trio left open. This piece is about what that gap is, how the LSM-on-a-lake design actually works, and the honest cases where Paimon is the right call and the larger number where it isn't.
What is Apache Paimon?
Apache Paimon is an open lake format that combines a Log-Structured Merge-tree with data-lake storage, purpose-built for building a real-time lakehouse where Flink and Flink CDC are first-class citizens. That one sentence carries the whole identity. It stores data as Parquet/ORC files in S3, HDFS, or any object store — so it's a lake format like the others — but it organizes those files as an LSM-tree, which is what makes it good at the thing the others struggle with: continuous, high-frequency, row-level upserts.
The lineage matters for understanding the design bias. Paimon began life inside the Apache Flink community as Flink Table Store, built to give Flink a native storage layer that could keep up with streaming writes and serve streaming reads. So where Iceberg's mental center of gravity is "a table you scan," Paimon's is "a table you continuously mutate and tail." It's the only one of the major open formats built on an LSM-tree, and nearly everything distinctive about it follows from that single choice.
Why does an LSM-tree matter for a lake format?
The LSM-tree is the reason Paimon can absorb a stream of updates without collapsing, and it's worth being precise about the mechanism because it's the entire pitch. An LSM-tree never updates data in place. Writes accumulate in an in-memory buffer, get flushed to sorted, immutable files, and those files are periodically merged in the background into larger, cleaner ones. Reads merge across the levels on the fly. It trades read-time work for write-time speed — which is exactly the trade you want when the write side is a relentless CDC stream and the read side can afford to do a little merging.
flowchart TB
CDC["Flink CDC stream
inserts · updates · deletes"] --> MEM["Write buffer
sorted by primary key, in memory"]
MEM -->|"flush"| L0["Level 0 files
small, recent, on object storage"]
L0 -->|"background compaction"| L1["Level 1+ files
larger, merged, fewer"]
L1 --> READ["Merge-on-read
latest value per key wins"]
L0 --> READ
READ --> Q["Query engines
Flink · Spark · Trino · StarRocks"]
Writes land cheaply in sorted level-0 files; compaction quietly merges them into fewer, bigger files over time; reads reconcile across levels so the newest value for each key wins. It's the RocksDB playbook, running on S3 instead of local SSD — which is both the clever part and the source of the caveats.
This is why Paimon shrugs off a workload that makes Iceberg wince. Because writes go into an ordered structure keyed by the primary key, an upsert doesn't require rewriting a large data file (Iceberg copy-on-write) or accumulating unbounded delete files that reads must reconcile (the naive merge-on-read failure mode). It requires appending a small sorted file and letting compaction clean up later. Paimon supports several merge engines on top of this — deduplicate (keep the latest), partial-update (merge columns from different streams into one row), and aggregation (roll up on write) — all of which are LSM-native operations rather than bolted-on rewrites.
How is this different from Iceberg, Hudi, and Delta?
The short version: Iceberg, Delta, and Hudi are batch-first formats with streaming capabilities added; Paimon is a streaming-first format that also does batch. That framing predicts almost every practical difference. Iceberg and Delta optimize for large-scale analytical scans and treat frequent mutation as something to be managed carefully. Hudi sits closest to Paimon philosophically — it was built for upserts and has a merge-on-read mode — but it's copy-on-write/merge-on-read over base+log files rather than a true LSM-tree, and its center of gravity is still batch incremental processing. Paimon is the one that put the LSM-tree in the lake.
| Apache Paimon | Apache Iceberg | Apache Hudi | Delta Lake | |
|---|---|---|---|---|
| Core structure | LSM-tree on object storage | Snapshots over immutable data files | Base + log files (CoW / MoR) | Parquet + transaction log |
| Built for | Streaming upserts first | Large analytical scans first | Upserts / incremental first | Batch + Spark first |
| High-frequency CDC upserts | Native strength | Workable but file-explosion prone | Good | Workable |
| Flink integration | First-class (its origin) | Good | Good | Improving |
| Streaming reads (tail the table) | Native — changelog producer | Incremental, less fluid | Incremental | Change data feed |
| Ecosystem breadth | Newest, growing | Broadest engine + catalog support | Broad | Broad (Databricks-centric) |
I want to be careful not to oversell here, because "newest and most specialized" is not the same as "best default." Iceberg's advantage — the reason I covered the catalog wars and its internals at length — is ecosystem gravity: nearly every engine, every cloud, every catalog speaks Iceberg now. Paimon's ecosystem is real and growing but younger. Choosing Paimon means choosing a narrower, deeper tool, and that's only the right trade when the depth is exactly what your workload needs.
Doesn't picking Paimon strand my data from the Iceberg ecosystem?
This was my biggest hesitation, and Paimon's answer to it is the feature that moved it from "interesting" to "deployable" for me: Iceberg compatibility. Paimon can be configured to generate Iceberg-compatible snapshots alongside its own LSM storage, so a table you write and mutate as a Paimon LSM-tree is simultaneously readable by any engine that speaks Iceberg — Trino, Spark, Athena, StarRocks — with the real-time data and its deletion vectors reflected. You get LSM write performance on the ingest side and Iceberg's read ecosystem on the query side, which is close to having it both ways.
The mental model that made it click for me: think of Paimon as the write-optimized front of a lakehouse table and Iceberg as the read-optimized face it can present. Flink CDC hammers the LSM front; analysts and BI tools query the Iceberg face. You're not choosing Paimon instead of Iceberg so much as choosing Paimon as the ingestion engine for tables that need heavy mutation, while keeping them legible to the Iceberg world everyone else lives in.
What are the trade-offs I'd actually worry about?
Every LSM-tree makes the same devil's bargain, and running one on object storage adds a twist, so here's what I'd stress-test before committing.
Compaction is a first-class operational concern, not an afterthought. The LSM design's write speed is borrowed against future compaction work. If compaction can't keep up with the write rate, level-0 files pile up, read amplification climbs, and query latency degrades — the exact HBase-flashback failure mode from my opening, now on your terms rather than despite them. Paimon gives you dedicated and inline compaction options and tuning knobs, but you own capacity-planning that background work. Budget compute for it; don't discover it under load.
Merge-on-read costs the reader. Freshly written data that hasn't been compacted yet must be merged at query time. So the more real-time you push the write side, the more merging the read side does until compaction catches up. There's a genuine freshness-versus-read-latency dial here, and where you set it is a workload decision, not a default.
Object-storage semantics matter. LSM-trees were designed for local SSDs with cheap random I/O; S3 has higher latency, request costs, and only eventual consistency guarantees on some operations. Paimon is engineered around this, but small-file overhead and request-cost amplification on a chatty write workload are real, and they're the kind of thing that shows up on the bill before it shows up in a benchmark.
Don't reach for Paimon if your workload is append-mostly analytics. The entire value of the LSM-tree is absorbing frequent row-level mutation. If you're landing large batches of mostly-immutable events for scan-heavy analytics — logs, clickstream, telemetry — you're paying for machinery you won't use, and Iceberg or Delta will give you a broader ecosystem with a simpler operational story. Paimon earns its keep specifically when the table is genuinely mutable at high frequency: CDC mirrors of operational databases, dimension tables that change constantly, wide rows assembled from partial updates across several streams. Match the tool to the mutation rate, not to the hype.
What does a Paimon pipeline look like in practice?
The canonical use case is a Flink CDC job mirroring an operational database into a Paimon primary-key table, kept seconds-fresh and immediately queryable. Because Paimon speaks Flink SQL natively, the whole thing is declarative — no custom operators, no hand-rolled upsert logic:
-- A Paimon primary-key table with deduplicate merge:
-- the latest row per order_id always wins, upserts handled by the LSM.
CREATE TABLE paimon.ops.orders (
order_id BIGINT,
customer_id BIGINT,
status STRING,
amount DECIMAL(12,2),
updated_at TIMESTAMP(3),
PRIMARY KEY (order_id) NOT ENFORCED
) WITH (
'bucket' = '4',
'merge-engine' = 'deduplicate',
'changelog-producer' = 'input',
'metastore.iceberg.enabled' = 'true' -- present an Iceberg face for readers
);
-- Stream a CDC source straight in. Flink keeps it live; Paimon absorbs
-- the upserts into the LSM-tree and compacts in the background.
INSERT INTO paimon.ops.orders
SELECT order_id, customer_id, status, amount, updated_at
FROM mysql_cdc_source.orders;The changelog-producer setting is the streaming-native touch worth noticing: Paimon can emit a proper changelog, so a downstream Flink job can tail this table as a stream of inserts/updates/deletes and build the next layer on top of it. That's the "streaming reads" property the batch-first formats approximate awkwardly — here it's the point of the design. You can chain Paimon tables into a streaming medallion architecture where each layer tails the one below it, which is genuinely hard to do cleanly on Iceberg.
What to carry away
Apache Paimon is the answer to a specific, real problem: the open lakehouse formats were built for batch and made streaming upserts feel like swimming upstream, and Paimon inverts the design by putting an LSM-tree on the lake so continuous mutation is the thing it's best at rather than the thing it tolerates. Reach for it when your table is mutated at high frequency by Flink CDC and you want real-time freshness without an Iceberg small-file nightmare — and lean on its Iceberg compatibility so choosing it doesn't exile your data from the ecosystem everyone else queries. Don't reach for it for append-heavy analytical scans, where you'd be buying LSM machinery you won't use and giving up ecosystem breadth for nothing. And whatever you do, treat compaction as a capacity-planning problem from day one, because the LSM-tree's gift for absorbing writes is a loan against merge work you will eventually pay back — the only question is whether you planned for the payment or got surprised by it under load.