# Real-Time AML on Kafka and Flink: Lessons From a Mid-Sized Bank

The project brief sounded simple: replace an overnight batch AML screening job with something real-time. What it actually meant, six months in, was rebuilding transaction monitoring for a bank with a few million retail customers on top of a stateful streaming platform, while a regulator's examination clock kept ticking in the background and every design decision had to be defensible to someone who'd never heard of Kafka. This is the architecture we landed on, the numbers we sized it against, and the parts that went wrong before they went right.

I want to separate two things up front that get conflated constantly, including by people who should know better: **AML** (anti-money laundering — the regulatory obligation under the Bank Secrecy Act to detect and report suspicious activity, ultimately producing a **SAR**, a Suspicious Activity Report, filed with FinCEN) and **fraud detection** (protecting the bank and its customers from unauthorized or deceptive transactions — chargebacks, account takeover, card-not-present fraud). They share infrastructure and often share a team, but they have different regulatory drivers, different urgency profiles, and — this matters more than it sounds — different latency requirements. Conflating them in one system is exactly the mistake that cost us the most time.

## Why does AML need a different SLA than fraud blocking?

The first architectural decision, and the one I'd make loudest if I were doing this again: **card-authorization-time fraud scoring does not live in the same system as AML case generation.** Blocking a fraudulent card swipe at the point of authorization has a real-time SLA measured in low hundreds of milliseconds — the payment network is waiting on your answer before the transaction clears. A stateful Flink job doing a RocksDB state lookup, evaluating five rules, and possibly calling out to a model server cannot reliably hit that SLA at p99 under load, and pretending it can is how you end up either declining good transactions on timeout or quietly widening your latency budget until the "real-time" system isn't.

AML, by contrast, doesn't need to block anything in-flight. A suspicious pattern generates a case for an investigator to review, and regulatory guidance gives you days, not milliseconds. That's a fundamentally different latency budget, and it's the budget that lets you afford a stateful, rule-plus-ML pipeline instead of a lookup-table fast path. We ended up with two systems: a low-latency authorization-time fraud score (a separate, much simpler service, out of scope for this piece) and the Kafka/Flink pipeline below, which does AML case generation and the fraud patterns that don't need to block a live transaction — velocity abuse, account takeover indicators, structuring.

| Tier | What it does | Latency SLA | System |
| --- | --- | --- | --- |
| Authorization-time fraud | Block/allow a card transaction in-flight | <300ms p99 | Separate low-latency scoring service |
| Near-real-time fraud | Flag account takeover, velocity abuse for review/step-up auth | <5s p95 | Kafka + Flink (this pipeline) |
| AML case generation | Detect structuring, layering, sanctions hits; open investigator case | <15 min p95 | Kafka + Flink (this pipeline) |
| Batch reconciliation | Reprocess corrected/late core-banking feeds nightly | Next business day | Batch job over the same event log |

## What does the pipeline actually look like?

The core banking system publishes every posted transaction to a Kafka topic, keyed by account ID so that all of one account's transactions land on the same partition and arrive at Flink in order — this ordering guarantee is not optional, since several of the rules below depend on seeing an account's transaction sequence correctly. Flink consumes with a `KeyedProcessFunction`, keeping a rolling window of recent transaction history per account in state, evaluates a set of rules and a model score against each incoming event, and emits alerts to a downstream topic that feeds case management. A parallel path persists the state snapshot that produced each alert to an immutable audit store, because "why did this fire" has to be answerable to an examiner months later, not just visible in a dashboard today.

```mermaid
graph TD
    CORE["Core banking system"] -->|"posted transactions"| KTOPIC["Kafka: transactionskeyed by account ID"]
    KTOPIC --> FLINK["Flink KeyedProcessFunctionper-account rolling state (RocksDB)"]
    FLINK --> RULES["Rule enginestructuring, velocity, geo, layering"]
    FLINK --> MODEL["ML risk scorefeature store + model server"]
    RULES --> COMBINE["Composite risk score"]
    MODEL --> COMBINE
    COMBINE -->|"above threshold"| ALERTS["Kafka: alerts"]
    COMBINE -->|"always"| AUDIT["Immutable audit logfeatures + score + rule versions"]
    ALERTS --> CASE["Case managementinvestigator queue → SAR filing"]
```

*Every alert writes twice: once to the queue an investigator actually works, once to an audit log that has to survive independently of Flink's own state, because state gets cleaned up on a TTL and the audit trail can't.*

## What rules actually catch known AML typologies?

Hard rules catch known patterns — the typologies AML examiners and FinCEN guidance have documented for decades. We ran five in production, each keyed and windowed per account:

| Pattern | What it detects | Why it matters |
| --- | --- | --- |
| **Structuring / smurfing** | Multiple transactions just under $10,000 to the same account within a rolling 24-hour window | Evading the Currency Transaction Report threshold is itself a federal crime, independent of the underlying funds' source |
| **Velocity abuse** | Transaction count or dollar volume exceeding a per-account baseline within a short window (e.g. >15 transactions/hour, well above that account's historical norm) | Compromised-account and mule-account activity both spike volume fast |
| **Rapid pass-through** | Large incoming credit followed by >80% withdrawal within 24 hours | Classic mule-account signature — money moves through, doesn't stay |
| **Geographic impossibility** | Two transactions whose implied travel speed (Haversine distance / time delta) exceeds any plausible mode of transport | Strong account-takeover signal, cheap to compute, very low false-positive rate |
| **Layering (short-chain)** | Funds moving through 2–3 internal accounts in quick succession before an external transfer | A documented laundering technique — obscure the source by adding hops |

We implemented the sequence-sensitive patterns (rapid pass-through, layering) with Flink's SQL `MATCH_RECOGNIZE` clause rather than hand-rolled state machines in the process function — it reads close to the regulatory language a compliance analyst would use to describe the pattern, which mattered when the same query had to be reviewed and signed off by non-engineers.

```sql
SELECT account_id, first_txn_time, last_txn_time
FROM transactions
MATCH_RECOGNIZE (
    PARTITION BY account_id
    ORDER BY event_time
    MEASURES
        A.event_time AS first_txn_time,
        LAST(B.event_time) AS last_txn_time
    PATTERN (A B{4,})
    WITHIN INTERVAL '24' HOUR
    DEFINE
        A AS A.txn_type = 'CREDIT' AND A.amount > 5000,
        B AS B.txn_type = 'DEBIT'
) AS rapid_out;
```

## What does the ML layer add that rules can't catch?

Rules only catch what someone already wrote a rule for. A gradient-boosted model scoring each transaction against dozens of behavioral features — deviation from an account's own spending baseline, merchant category mix, time-of-day pattern, device/session signals where available — catches the drift and novel variations that a fixed threshold misses by design. The model didn't replace the rules; it produced a composite risk score alongside them, and that composite score is what actually triages the investigator queue, because a raw list of every rule firing, unranked, is how you get alert fatigue.

The operational reality nobody warns you about going in: **feature serving latency, not model inference, is usually the bottleneck.** A model that takes 5ms to score is worthless if computing its input features requires a database round-trip that takes 200ms per transaction at production volume. We ended up materializing the rolling behavioral features directly in Flink state alongside the rule state, so the same per-account window that feeds the rules also feeds the model — one state store, two consumers, instead of a separate feature-store round-trip on the hot path.

**Model governance is not optional once a model influences a regulated decision.** Any model contributing to AML case generation falls under the same model-risk-management expectations (documented model purpose, independent validation, ongoing performance monitoring, a defined retraining/retirement process) that examiners apply to any other quantitative model the bank relies on. Treating the fraud model like an internal ML side project — no validation record, no monitoring for score drift, nobody who can explain a specific score months later — is a real finding waiting to happen, not a hypothetical risk.

## How do you size Kafka and Flink for a mid-sized bank's real volume?

"Real-time" projects get sized against demo data and then fall over on day one of production load. Here's roughly what we sized against for a bank in the low millions of retail accounts:

| Metric | Typical | Peak (payday / holiday) |
| --- | --- | --- |
| Sustained transaction rate | ~180 TPS | ~850 TPS |
| Daily transaction volume | ~15M | ~40M (peak day) |
| Kafka partitions (transactions topic) | 32, keyed by account ID |
| Flink parallelism | 32, matched to partition count |
| Per-account state size (30-day rolling window) | 2–40 KB depending on account activity |
| Total operator state, steady state | ~90 GB across the cluster |
| Checkpoint interval | 30s (tuned down from 10s — see below) |

Ninety gigabytes of state doesn't sound large next to a data warehouse, but it's large for what has to live in Flink's managed state, accessed on every event, with sub-second p99 read/write latency per key. That number is the whole reason the next section exists.

## What actually broke in production, and what we changed

**The heap state backend that worked fine in every test environment fell over under real account-history volume.** Development and staging both ran Flink's default heap-based state backend, because it's faster for small state and nobody thought to test with three months of realistic per-account history loaded. In production, TaskManager heap pressure from tens of gigabytes of live state produced GC pauses long enough to blow through checkpoint timeouts, which cascaded into checkpoint failures, which cascaded into an ops page at 2 a.m. Migrating to RocksDB as the state backend — spilling state to local disk instead of holding it all in JVM heap — fixed the throughput problem but introduced a new one: RocksDB read/write latency is real, and a naive "hit RocksDB per rule per event" implementation added enough per-event overhead that we had to consolidate the rules to share a single state read per account per event, not five separate ones.

**Alert fatigue was the second production fire, and it was a tuning problem disguised as a technology problem.** Our initial thresholds, taken more or less from generic AML guidance without tuning against our own account base's actual behavioral baseline, produced far more alerts than the investigator team could review in a business day. The fix wasn't a smarter rule — it was ranking every alert by the composite ML risk score before it ever reached a human queue, so investigators worked the highest-confidence cases first instead of the chronologically-first ones. Alert volume didn't drop; the queue got triaged, and time-to-resolution on genuinely high-risk cases dropped by more than half.

**A downstream case-management outage taught us that consumer lag and per-account ordering don't mix well during replay.** When the system consuming our alerts topic went down for several hours, Kafka did exactly what it's supposed to do — retained the backlog — but replaying hours of backed-up alerts after recovery meant Flink was reprocessing under load spikes well above steady state, and we had to be careful that catch-up processing didn't starve real-time events on the same keyed operators. We ended up prioritizing live traffic over backlog replay explicitly, rather than assuming FIFO replay would just sort itself out.

**Late-arriving corrections from the core banking system's own batch reconciliation nearly caused a silent miss.** Occasionally a transaction posted hours earlier gets corrected — a reversal, a category change — and that correction arrives as a new event well outside any reasonable watermark for the original transaction's window. A structuring pattern that only becomes visible once the correction lands can be missed entirely if your watermark strategy just drops late events. We added a secondary, coarser-grained batch reconciliation pass specifically to catch patterns that only resolve once corrected data is in, rather than trying to solve it by widening the streaming watermark indefinitely and paying the latency cost on every event for a rare case.

## What does disaster recovery actually require here?

DR for a stateful streaming AML pipeline is a different problem than DR for a stateless service, and the gap between them is exactly the amount of state you're carrying. Our targets: an RPO of a few minutes on transaction data (Kafka replicated cross-region, acceptable to lose at most a few minutes of un-replicated events in a true regional failure) and an RTO of a few hours for the full pipeline. The RTO number sounds generous until you've actually timed a cold restart of a Flink job restoring ninety gigabytes of RocksDB state from a savepoint — that restore time scales with state size, not with how urgently you need the system back, and testing failover under realistic state volume (not empty-state failover, which tells you nothing) is the only way to find out your actual number before an examiner or an outage finds it for you.

**Test failover with production-representative state, not a clean slate.** An empty-state Flink job restarts almost instantly and tells you nothing about your real RTO. Load a savepoint with realistic state volume during your DR test, every time — that's the number that will actually matter during a real incident.

## What to carry away

Separate AML/near-real-time fraud from authorization-time fraud blocking — they have genuinely different latency budgets, and one pipeline trying to serve both ends up compromising on both. Size Kafka partitions and Flink parallelism against your real peak volume, not a demo, and budget for RocksDB from day one if your state will ever hold more than a trivial rolling window — heap-backed state is a trap that only shows itself under real production load. Rules catch known typologies; a composite ML score is what actually makes the investigator queue usable, not a replacement for the rules. And treat the audit trail — the features and rule versions that produced a given alert — as a first-class, independently durable output, because state gets cleaned up on a TTL and a regulator's question won't arrive on your schedule.

For the streaming mechanics underneath all of this — state backends, checkpointing, watermarks, exactly-once — see [Apache Flink Internals](apache-flink-internals). For the Kafka side of partitioning and ordering guarantees, see [Kafka Internals](kafka-internals). And for the broader question of when Flink is the right choice over alternatives, [Flink vs Kafka Streams vs Spark Structured Streaming](stream-processing-flink-kafka-streams-spark) covers that trade-off in more depth than fits here.