Change data capture demos beautifully. Point a connector at a database, watch rows appear in a topic, and everyone in the room understands immediately why this is better than the nightly batch job it replaces. The proof of concept takes an afternoon. Then you run it for five years, and it teaches you things.
I've now stood up CDC pipelines on Postgres, MySQL, SQL Server and Oracle, in shops that ranged from careful to chaotic, and the pattern is consistent: almost nothing that goes wrong with CDC is about the change stream itself. Steady-state streaming is the part that works. What breaks is the initial load, the boundary between the source database's operational reality and your pipeline's assumptions, and the day you need to do it all again. I've covered how log-based capture works in the Debezium article — this is what happens after the demo.
Scar one: the initial snapshot
The hardest part of streaming a table is the first read of it. Before a connector can stream changes it needs a consistent starting point, which classically means reading the whole table while holding enough of a lock or a consistent view for the result to make sense. On a 4TB table with an active write workload, that is not a background detail — it's the thing that makes a DBA revoke your access.
What actually goes wrong, in order of how often I've seen it:
- The snapshot never finishes. It takes eleven hours, the connector restarts on hour ten for an unrelated reason, and it starts over from the beginning. Meanwhile the log it needs to resume from is aging out.
- The snapshot blocks writers. On engines and configurations where a consistent snapshot takes locks, a large table can stall production traffic — and you will find out from the application team, loudly.
- The snapshot succeeds and nobody can tell. Downstream consumers see a partially-loaded table for hours with no signal that it's incomplete, and someone builds a report on it.
The defaults I now insist on: use incremental snapshotting where the connector supports it — chunked reads interleaved with live changes, so a restart resumes rather than restarts and writers aren't blocked for hours. Snapshot one table at a time for anything large, not the whole schema in one job. And publish an explicit readiness signal per table so downstream consumers know when a dataset is trustworthy rather than inferring it from row counts.
Scar two: the replication slot that filled the disk
This is the incident that gets CDC banned at a company, so it deserves its own section. On Postgres, a replication slot guarantees the database keeps write-ahead log segments until the consumer has confirmed them. That guarantee is the feature. It is also a loaded weapon: a consumer that stops consuming makes the source database retain WAL forever.
The chain is always the same. Your connector dies at 6pm on a Friday — a bad deployment, an expired credential, a Kafka issue. The slot stays. WAL accumulates. Saturday morning the primary's disk fills, and a full disk on a primary is not a data-pipeline incident any more, it's a production outage with your name on it. MySQL and SQL Server have their own versions of this shape (binlog retention, CDC capture tables that grow when the cleanup job can't keep up); the mechanism differs, the lesson doesn't.
Three controls, all boring, all mandatory: alert on replication slot lag in bytes with a threshold well under free disk, and treat it as a page rather than a ticket; set a maximum slot retention where your engine supports it, accepting that a badly-lagged consumer gets invalidated instead of taking the database down — a broken pipeline is a much better outcome than a broken primary; and give the connector a dead-man's handle, so a consumer that has been down beyond your recovery window drops its slot deliberately and re-snapshots rather than holding the database hostage.
graph TB
DB[("Source database")] -->|"WAL / binlog"| SLOT{"Replication slot
retains until confirmed"}
SLOT --> CONN["CDC connector"]
CONN -->|"heartbeat every N sec"| DB
CONN --> TOPIC[("Kafka topics
one per table")]
TOPIC --> SINK["Sinks: warehouse, lake, services"]
SLOT -.->|"consumer stops"| GROW["WAL grows
unbounded"]
GROW -.-> DISK["Primary disk fills
= production outage"]
MON["Monitor: slot lag in BYTES"] --> SLOT
MON -->|"threshold breach"| PAGE["Page — not a ticket"]
HB["Heartbeat table
writes a row on a timer"] --> DB
HB -.->|"advances the slot on
low-traffic databases"| SLOT
The heartbeat is the trick nobody mentions in the getting-started guide. On a quiet database — or one where the tables you capture are quiet while others are busy — the connector has nothing to confirm, so the slot's confirmed position never advances and WAL retention grows even though your pipeline is perfectly healthy. A tiny table written on a timer keeps the position moving and turns a mysterious growth curve into a non-event.
Scar three: schema drift, the hard way
Someone adds a column. In a good week your pipeline picks it up and life continues. The bad weeks are more interesting:
| Source change | What tends to happen | What to do about it |
|---|---|---|
| Add a nullable column | Usually fine — new field appears downstream | Nothing, if your sink evolves schemas |
| Drop a column | Sink rejects, or silently keeps a stale field forever | Registry compatibility rules; treat as a contract change |
| Rename a column | Reads as drop + add: the old data does not follow | Two-phase change at the source — add, backfill, cut over, drop |
| Type change (int → bigint, varchar widening) | Downstream parse failures, or silent truncation | Compatibility mode set to reject, so it fails loudly at the boundary |
| Primary key change | Message keys change → partitioning and compaction semantics break | Re-snapshot the table. There is no clever alternative. |
| Table truncated or swapped | Depending on engine, no events at all — downstream keeps stale rows | Detect out of band; treat truncate as a re-snapshot trigger |
The two rows in bold are where real incidents live. A primary-key change is a genuine re-snapshot, and pretending otherwise produces a compacted topic containing two identities for the same row. Truncate is worse because it is silent on several engines: the table empties and your warehouse copy doesn't, so it stays confidently wrong until someone reconciles counts.
The structural fix isn't technical. CDC makes your source database's schema a public API, and nobody told the team that owns it. A schema registry with compatibility enforcement gives you a mechanical gate; a written agreement with the owning team about what constitutes a breaking change is what stops the gate from being hit at 2am. That's the argument for a registry with real compatibility rules, and for treating this as a producer relationship rather than a tapping arrangement.
Scar four: deletes, and the tombstones nobody handles
Every CDC tutorial covers inserts and updates. Deletes are where correctness quietly dies.
A delete arrives as an event whose payload is mostly empty — the row's identity plus a marker — and, on compacted topics, is typically followed by a tombstone (a null-valued record) so log compaction can eventually forget the key entirely. Two consequences catch people out. Sinks that treat a null payload as "malformed message, skip it" will happily skip your deletes, leaving rows in the warehouse that no longer exist in the source. And soft deletes — the row is still there with is_deleted = true — never produce a delete event at all; they're updates, and every downstream consumer needs to know that a filter is required.
What I do now: land deletes explicitly as an operation column rather than dropping rows on arrival, so a warehouse table can be reconstructed at any point in time and an accidental delete flood is recoverable. Then, if consumers want a current-state view, that's a view over the change log rather than a destructive apply.
-- Land the change log, then derive current state. The extra table is worth it:
-- a bad delete flood becomes a recoverable event rather than a restore-from-backup.
create table raw.customers_changes (
op char(1) not null, -- c=create, u=update, d=delete, r=snapshot read
customer_id bigint not null,
payload variant, -- the "after" image; null for deletes
source_ts timestamp_ntz not null, -- from the DB's log, NOT ingestion time
ingest_ts timestamp_ntz not null default current_timestamp(),
source_lsn string -- log position: the real ordering key
);
-- Current state as a view: last change per key wins, deletes excluded.
-- Order by the log position, never by wall clock — two changes can share a
-- millisecond, and clocks are not a total order across a failover.
create or replace view analytics.customers as
select payload:*
from (
select *, row_number() over (
partition by customer_id
order by source_lsn desc) as rn
from raw.customers_changes
)
where rn = 1 and op != 'd';The 3am problem: reprocessing with no plan. Every CDC pipeline eventually needs to be replayed — a sink bug corrupted a month of data, a transformation was wrong, a table needs to move to a new schema. The question that decides how bad that night is: can you rebuild a target from the change log alone, or do you need the source database again? If your topics have infinite retention or you land raw changes in object storage, replay is a job you run. If they're on seven-day retention and you dropped the raw layer to save money, replay means re-snapshotting a production database under time pressure, which is exactly the operation the DBA revoked your access over the first time. Two cheap insurances I'd never skip: land the raw change log to cheap storage before any transformation, indefinitely — this is the difference between a replay and an outage; and make every sink idempotent on (key, log position), so re-running the same changes twice is a no-op rather than a duplicate hunt. The teams that get paged at 3am and go back to sleep by 3:30 are the ones that did both before they needed them.
Scar five: ordering, and the lies of wall-clock time
Two more that cost me real days. Ordering is per key, not global. Changes to a single row arrive in order because they share a partition; changes across tables do not, which means a naive consumer can see a child row before its parent exists. Design sinks to tolerate that — upsert without foreign-key enforcement at the landing layer, resolve relationships downstream — rather than assuming a global order that the system never promised.
And use the source's log position for ordering, not timestamps. Source clocks drift, batch transactions can share a millisecond, and after a failover the new primary's clock may disagree with the old one's. The log sequence number is the only total order you're given; the timestamp is a helpful annotation. Every deduplication bug I've debugged in a CDC pipeline traced back to someone sorting by updated_at.
Operational defaults I'd start any new pipeline with
- Alert on slot/binlog lag in bytes and on connector liveness, as pages. These two are the difference between an inconvenience and a database outage.
- Heartbeat table written on a timer, so quiet databases still advance the confirmed log position.
- Incremental snapshots, one table at a time, with a per-table readiness signal downstream consumers can check.
- Raw change log landed to cheap storage before transformation, retained far longer than feels necessary. It is your replay button.
- Idempotent sinks keyed on (primary key, log position), so replay is safe by construction.
- Schema registry with compatibility enforcement, plus a written agreement with the owning team about breaking changes.
- A reconciliation job — nightly row counts and a checksum on a sample of keys, source versus target. It catches the silent classes (skipped tombstones, missed truncates) that no lag metric will ever show you.
That last one is the most under-appreciated control in this article. Lag metrics tell you the pipeline is moving; only reconciliation tells you it's right. Every silent CDC bug I've found in five years was found by a count comparison, not by a monitor on the stream.
What to carry away
Steady-state CDC works; the edges are where you'll spend your operational life. Plan the initial snapshot as a project of its own — incremental, one table at a time, with an explicit readiness signal. Treat the replication slot as a loaded weapon: alert on lag in bytes, cap retention where you can, heartbeat quiet databases, and accept a broken pipeline over a full primary disk.
Handle schema drift as a contract problem, not a parsing problem, with a registry enforcing compatibility and a real agreement with the team that owns the source — because CDC turns their schema into your API whether or not anyone said so. Handle deletes and tombstones explicitly, land the operation rather than applying it destructively, and remember that soft deletes never produce delete events at all.
Then buy the two insurances that make the bad night survivable: land the raw change log to cheap storage so replay is a job rather than an emergency, and make sinks idempotent on log position so replay is safe. Order by log sequence, never by timestamp. And run a reconciliation job, because the failures that hurt most are the silent ones, and no lag dashboard has ever caught one.