Start here

205 articles is more than anyone wants to browse, and a reverse-chronological list is a terrible way in — the newest thing I wrote is rarely the thing you need. So this page is organised by the question you actually arrived with. Pick the one that matches, read three or four pieces, ignore the other two hundred and one.

  • 205deep-dives
  • 2018writing since
  • 18–22min, typical read

“I'm choosing a data platform and everyone is lying to me”

The most consequential decision on this list, and the one with the worst signal-to-noise ratio. Read the assessment first — it's scored, and it tells you which axes actually decide it.

  1. Snowflake vs Databricks vs Microsoft FabricTen axes, scored. The honest finding: all three are Adopt and the decision is mostly not technical.
  2. Snowflake vs BigQuery vs RedshiftThe fourth option the assessment leaves out, and how the pricing models really differ.
  3. The Iceberg REST Catalog WarsWhy the catalog — not the engine — is the lock-in decision you're actually making.
  4. Databricks InternalsIf you want to know what you're buying rather than what the deck says.

“I need to put an AI agent in production”

Demos are easy and production is not. These are in the order I'd actually tackle the problems.

  1. Agent Harness DesignThe scaffolding around the model is the product. Start here before picking a framework.
  2. Evaluating LLM Agent SystemsBuild the eval harness before the agent. Everything else is guessing.
  3. Prompt Injection & GuardrailsThe attack that gets skipped until it's an incident. Tool arguments are untrusted input.
  4. Fine-tuning vs RAG vs PromptingMost teams reach for fine-tuning when they have a retrieval problem.
  5. Building a Voice AgentIf it's voice: the latency budget decides the architecture, not the model.

“My pipelines keep breaking and I'm on call”

The unglamorous cluster, and the one I'd read first if you inherited someone else's platform.

  1. Data Pipeline On-CallRecovery ergonomics matter more than authoring ergonomics. You author once and recover forever.
  2. Data ContractsUpstream broke you on a Friday. A contract only works if you'll break their build.
  3. Data ObservabilityDetecting the failure you didn't write a test for.
  4. Airflow vs Dagster vs PrefectScored. Includes the migration cost nobody puts in the business case.

“I want to understand how these systems actually work”

Internals, for their own sake. These are the pieces I most enjoyed writing, and the ones that hold up best years later.

  1. ClickHouse InternalsHow a columnar engine scans billions of rows a second. Sparse indexes and merges.
  2. BigQuery Internals: Dremel & ColossusFrom 2019 and still the clearest explanation of separating storage from compute.
  3. Apache Flink InternalsCheckpoints, watermarks, and what exactly-once really costs.
  4. Kafka in ProductionThe patterns that survive contact with a real incident.

“I'm building the practice, not just the pipeline”

For leads and architects deciding how a team should work rather than what it should install.

  1. The Data Engineering LifecycleThe frame everything else on this site sits inside.
  2. Analytics Engineering with dbtSQL transformations are software. The practice, not the tool.
  3. Data Platform Techniques, scoredMedallion, metrics layer, contracts, data mesh — including how badly each fails when nobody maintains it.
  4. Data Mesh: What Actually WorksThe diagnosis is right; the prescription is a reorg you can't ship. Field notes.
  5. The Metrics Layer & Headless BIWhen every dashboard shows a different revenue number.

“I have a compliance problem wearing a data costume”

Regulated industries — health, life sciences, finance — where the architecture is downstream of the auditor.

  1. PII Tokenization & PrivacyWhy an unsalted hash of an identifier is not de-identification.
  2. GA4GH Genomic Data SharingStandards-based sharing when the data can never be centralised.
  3. FHIR Streaming into SnowflakeClinical data in near-real-time without losing the audit trail.

Prefer to browse? The full archive has search, tag filters and a year-by-year timeline; Topics groups everything into 16 themes. The Architecture Radar is where the opinions get scored, and its methodology explains how — including the biases I know I have.