The question that ends every lakehouse presentation, usually from the most senior engineer in the room and usually in a tone of polite disbelief: *"So it's just Parquet files in a bucket. How exactly do you get ACID out of that?"* It's the right question. Object storage gives you almost nothing to build transactions on — no locks, no multi-object atomic writes, historically not even a consistent directory listing. And yet the claim is that a directory of files on S3 supports concurrent readers and writers, snapshot isolation, and time travel.

The answer is a log, and it's a genuinely elegant piece of engineering that is also less magic than the marketing suggests. I covered [the lakehouse architecture and why it exists](lakehouse-architecture-delta-lake) a couple of weeks ago at the level of "what problem does this solve." This is the layer underneath: what's actually in _delta_log, how a commit becomes atomic on a storage system that offers you no atomicity, and — the part that doesn't make it onto the slide — which of Delta's headline performance features you cannot have unless you're paying Databricks.

## What's actually in a Delta table?

A Delta table is a directory of Parquet data files plus a _delta_log subdirectory containing an ordered sequence of JSON commit files that define which of those Parquet files are currently part of the table. That last clause is the whole trick: **the log is the table**. The Parquet files are just bytes on disk, and a file's presence in the directory means nothing. It's in the table if and only if the log says so.

```text
my_table/
├── part-00000-a1b2....snappy.parquet     <- just bytes. Not "in" the table
├── part-00001-c3d4....snappy.parquet        unless the log says it is.
├── part-00002-e5f6....snappy.parquet
└── _delta_log/
    ├── 00000000000000000000.json          <- commit 0: the table's whole history
    ├── 00000000000000000001.json          <- commit 1
    ├── 00000000000000000002.json
    └── ...
```

Each numbered JSON file is one atomic commit, containing a set of **actions**. The important ones are simple to the point of being anticlimactic:

| Action | What it means |
| --- | --- |
| add | This Parquet file is now part of the table — with its size, partition values, and min/max column statistics |
| remove | This file is no longer part of the table (a tombstone — the file still physically exists) |
| metaData | Schema, partition columns, table properties |
| protocol | Minimum reader/writer versions required to safely touch this table |
| commitInfo | Provenance — who did what, when, which operation |

An UPDATE that changes one row doesn't modify a Parquet file, because Parquet files are immutable. It writes a *new* file with the corrected data and commits {remove: old_file, add: new_file} as one atomic action set. The old file is still sitting in the bucket, untouched — which is exactly why time travel is nearly free, and exactly why your storage bill grows even when your table doesn't.

## How does a commit become atomic?

The commit is the **creation of the next numbered JSON file**, and its atomicity is borrowed entirely from the storage system's ability to guarantee that only one writer can create a given filename. That's it. That's the mechanism. If two writers both read version 4 and both try to write ...005.json, exactly one must win.

This is where the elegance meets reality, because storage systems differ on whether they'll give you that guarantee. HDFS has atomic rename, so it's straightforward — the [NameNode](hadoop-hdfs-internals) is a real metadata service with real mutual exclusion. Azure Data Lake Storage offers what's needed. **S3 is the problem child**: it has no put-if-absent, no atomic rename (a "rename" is a copy plus a delete), so two writers can both happily create the same key and the last one silently wins. Delta's answer is the LogStore abstraction — a pluggable layer where the S3 implementation brings in an external coordinator (DynamoDB, in the multi-writer setup) purely to arbitrate who gets to claim the next version number.

**The mental model worth keeping:** Delta didn't invent transactions on object storage. It reduced "transactions on object storage" to "atomically create one file with a specific name," and then went shopping for that one primitive on each storage system. Where the storage provides it, Delta is elegant. Where it doesn't — S3 — you need a coordinator bolted on, and the single-writer-per-table default that everyone quietly runs with is not a coincidence. When someone tells you the lakehouse is storage-agnostic, this is the asterisk.

### Optimistic concurrency, concretely

Delta uses optimistic concurrency control: assume conflicts are rare, do the work, and validate at the last moment. A writer reads the current version, writes its data files, then attempts to commit. If someone else claimed the version number first, it doesn't just fail — it re-reads what changed and asks whether that change actually conflicts with what it did. Two writers appending to different partitions don't conflict, so the loser simply retries against the new version. Two writers updating the same files do, and one of them raises.

```mermaid
sequenceDiagram
    participant W1 as Writer A
    participant LOG as _delta_log
    participant W2 as Writer B
    W1->>LOG: read — current version 4
    W2->>LOG: read — current version 4
    W1->>W1: write new Parquet files
    W2->>W2: write new Parquet files
    W1->>LOG: create 005.json
    LOG-->>W1: won — commit 5
    W2->>LOG: create 005.json
    LOG-->>W2: lost — file exists
    W2->>LOG: re-read 005: did it touch my files?
    LOG-->>W2: different partition — no conflict
    W2->>LOG: create 006.json
    LOG-->>W2: won — commit 6
          
```

Losing the race isn't losing the write. Writer B re-reads the winning commit, checks whether it actually conflicts, and retries at the next version — so concurrent appends to different partitions both succeed. The data files B already wrote are still valid; only the commit is retried. This is why "optimistic" is the right word: the expensive work is done before you find out.

## Doesn't reading get slow after a million commits?

It would, and this is where checkpoints come in. Reconstructing the table state means replaying every commit from zero — fine at version 12, absurd at version 120,000. So every 10 commits Delta writes a **Parquet checkpoint** that summarizes the entire state up to that point. A reader finds the most recent checkpoint, loads it, and replays only the handful of JSON commits after it.

Checkpoints being Parquet rather than JSON matters more than it looks: the reader can use columnar projection and predicate pushdown on the checkpoint itself. On a table with a million files, "give me the add actions and their min/max stats" is a columnar scan, not a JSON parse of a million records. It's the same argument I made about columnar formats in the [Parquet and ORC internals](parquet-orc-internals) piece, applied recursively to the metadata.

### Data skipping falls out of the log for free

Remember that each add action carries min/max statistics per column. That means the log — which the query engine has already loaded — knows the value range of every file before touching any data. A query with WHERE order_date = '2021-04-15' consults the log, discards every file whose min/max range excludes that date, and never issues a read for them.

This is why Delta on object storage can outperform the naive Hive-style setup it replaced. Hive pruned by *directory*, so skipping required your predicate to align with your partition columns. Delta prunes by *file statistics*, so it can skip on any column it has stats for. You get partition-pruning-like benefits on columns you never partitioned by, and the mechanism costs one metadata read you were doing anyway.

## Time travel, VACUUM, and the tension between them

Time travel is almost a side effect. Reading version 7 means replaying the log to commit 7 and reading exactly the files it says were live then — and since nothing is ever physically deleted at commit time, those files are still there.

```sql
-- Both of these are just "replay the log to a point, read what it lists"
SELECT * FROM my_table VERSION AS OF 7;
SELECT * FROM my_table TIMESTAMP AS OF '2021-04-15';

-- Physically delete files that no commit references any more.
-- This is what actually reclaims storage — and what kills time travel.
VACUUM my_table RETAIN 168 HOURS;
```

And there's the tension nobody mentions until the invoice arrives. Every remove action is a tombstone, not a deletion — the bytes stay. A heavily-updated table accumulates dead files indefinitely, and its storage footprint bears no relationship to its row count. VACUUM is what reclaims that space, and it does so by destroying exactly the history that time travel depends on. Retention isn't a tuning knob; it's you deciding how far back you're willing to pay to be able to look. Set it too short and someone's VERSION AS OF audit query breaks; too long and you're paying to store every version of a table that gets rewritten hourly.

**Don't set VACUUM retention below your longest-running query.** A reader that resolved its file list from version 40 and is still scanning when a VACUUM deletes those files gets a FileNotFoundException from underneath a query that was perfectly valid when it started. The default 7-day retention looks wastefully conservative right up until you understand it's protecting long readers, not just your audit trail. The safety check that stops you from going below it exists because someone learned this the expensive way.

## The part that isn't on the slide

Here's the thing I'd want to know before betting an architecture on this, and it's the honest caveat that gets glossed over in every "open format, no lock-in" pitch: **the transaction protocol is open source, and the performance features that make it fast are not.**

Everything above — the log, the commit protocol, optimistic concurrency, checkpoints, min/max data skipping, time travel — is in open-source Delta Lake and works on plain Spark. But OPTIMIZE (compacting the small-file swamp that streaming writes inevitably produce) and ZORDER BY (multi-dimensional clustering that makes data skipping dramatically more effective) are **Databricks Runtime features**, not part of the OSS release. Run open-source Delta on your own Spark cluster and you get correctness with none of the file-layout management, which means you either write your own compaction or watch your table degrade into thousands of tiny files that no amount of min/max stats will save you from.

| Capability | Open-source Delta Lake | Databricks Runtime |
| --- | --- | --- |
| Transaction log, ACID, optimistic concurrency | Yes | Yes |
| Checkpoints, time travel, VACUUM | Yes | Yes |
| Min/max data skipping | Yes | Yes |
| **OPTIMIZE** (small-file compaction) | **No** — roll your own | Yes |
| **ZORDER BY** (multi-dim clustering) | **No** | Yes |

I want to be fair here rather than cynical: an open format with an open, documented commit protocol is genuinely valuable, and it's more openness than the proprietary warehouses offer. Your data remains readable Parquet with a readable log forever, and that is the lock-in that actually matters. But "open source" and "open source and equally good everywhere" are different claims, and in 2021 Delta is the first without being the second. If your plan is OSS Delta on your own [Spark](spark-internals-rdd-catalyst-tungsten) cluster, budget for building compaction yourself — and price that honestly against a Databricks bill rather than assuming the gap doesn't exist.

## What to carry away

Delta Lake's core insight is a reduction, not a magic trick: it turns "ACID on object storage" into "atomically create one numbered file," and then relies on the storage system to provide that single primitive — which is why S3 needs a coordinator and why the abstraction has an asterisk. The log *is* the table; Parquet files are inert bytes until a commit vouches for them, which is what makes updates cheap, time travel nearly free, and your storage bill grow independently of your row count until VACUUM reconciles them. Checkpoints keep reads from replaying history forever, and min/max stats in every add action give you file-level skipping on columns you never partitioned by. Just go in knowing where the open part ends: the protocol is yours, the correctness is yours, and in 2021 the file-layout management that keeps a Delta table fast — OPTIMIZE and Z-ORDER — is still something you either buy or build.