Before analytics engineering had a name, I inherited a "reporting layer" that was a 3,000-line stored procedure, four scheduled SQL scripts that each redefined revenue slightly differently, and a folder of Tableau extracts whose logic lived only in the workbooks. The CFO and the VP of Sales quoted different revenue numbers in the same meeting, both pulled from "the data warehouse," and the honest answer to "which one is right?" was that nobody could trace either back to a definition they could read. There was no version control, no tests, no lineage — just SQL accreting in a dozen places, each one load-bearing and none of them reviewed.
Analytics engineering is the discipline that made that era end. It's often reduced to "the dbt thing," which undersells it: the tool matters, but the shift is that the people transforming data started working like software engineers — version control, testing, modularity, code review, CI — on the layer that turns raw tables into trusted, well-modeled datasets. This is what the discipline actually is, the role it created, the layered model that gives it structure, the semantic layer that ends the dueling-revenue-numbers problem, and where it bumps into its own ceilings. I'll use dbt throughout because it's the tool that defined the practice — but the practice is the point.
What is analytics engineering?
Analytics engineering is the application of software-engineering practices — version control, automated testing, modularity, code review, CI/CD, and documentation — to the transformation of raw data into clean, reliable, well-defined datasets that analysts and BI tools consume. It's the layer between getting data into the warehouse and getting insight out of it, and before it existed that layer was done with no engineering rigor at all, which is why it was a constant source of wrong numbers.
The discipline produced a role: the analytics engineer, who sits between the data engineer and the data analyst. The data engineer moves and lands data (pipelines, ingestion, infrastructure). The analyst uses data to answer business questions. In between sat a gap — who turns the messy raw tables into trustworthy, modeled, documented datasets the analyst can rely on? — and that gap was usually filled badly, by whichever analyst was most comfortable with SQL, working without a net. The analytics engineer owns that transformation layer as a real software artifact.
Analytics engineer = the SQL fluency of an analyst plus the engineering discipline of a software developer. They don't build ingestion infrastructure and they don't build dashboards. They build and own the trusted modeled layer in between — the data models, the tests, the metric definitions — as version-controlled, tested, documented code.
Why it emerged: ELT and the cloud warehouse
The discipline didn't appear because someone wanted a new job title. It was forced by a technology shift. When cloud warehouses — Snowflake, BigQuery, Redshift — made storage and compute cheap and elastic, the old ETL pattern (transform data before loading, in a separate engine) flipped to ELT: load raw data into the warehouse, then transform it there with SQL. That one change relocated transformation into the warehouse, where the language is SQL and the people who are fluent in SQL are analysts, not necessarily software engineers.
So a large population of SQL-fluent, non-software-engineer people suddenly owned production transformation logic — and immediately hit every problem software engineering had already solved: no source control, no testing, no reuse, no environments, no way to review a change before it broke a dashboard. dbt's insight was to bring those solved practices to SQL transformation: write each transformation as a SELECT, let dbt handle the DDL and dependency order, and put the whole thing in Git with tests and docs. The discipline is "software engineering practices, finally applied to the analytics layer." The tool just made it accessible to people who think in SQL.
How dbt embodies the practice
The mechanics are worth a quick pass (the internals piece goes deeper). In dbt, a model is a SELECT statement in a .sql file; dbt wraps it in the right CREATE and figures out the rest. Models reference each other with ref(), and from those references dbt builds a DAG — so it knows the build order and the full lineage without you maintaining it by hand:
-- models/marts/finance/fct_revenue.sql
with orders as (
select * from {{ ref('stg_orders') }}
),
payments as (
select * from {{ ref('stg_payments') }}
)
select
o.order_id,
o.customer_id,
o.placed_at::date as revenue_date,
sum(p.amount_usd) as revenue_usd
from orders o
join payments p using (order_id)
group by 1, 2, 3The engineering practices fall out of this naturally. Everything is a file, so it lives in Git with branches, pull requests, and code review. Tests are declarative (not_null, unique, accepted_values, relationships) and run in the build. Jinja and macros let you write transformation logic once and reuse it (DRY), instead of copy-pasting the revenue calculation into twelve scripts. Documentation is declared alongside the model and rendered with the lineage graph. And dbt enforces environments — your dev changes build into a dev schema, never touching prod until merged. None of this is exotic to a software engineer; that's exactly the point.
The layered model: staging, intermediate, marts
The structural backbone of analytics engineering is a layered transformation, and getting these layers right is most of what separates a maintainable project from a swamp:
graph LR
SRC["Raw sources
(loaded by EL)"]
subgraph Staging["Staging (1:1 with sources)"]
STG["stg_orders, stg_payments
rename, cast, clean"]
end
subgraph Intermediate["Intermediate"]
INT["int_orders_joined
reusable business logic"]
end
subgraph Marts["Marts (business-facing)"]
DIM["dim_customers"]
FCT["fct_revenue"]
end
BI["BI / metrics / ML"]
SRC --> STG --> INT --> DIM
INT --> FCT
DIM --> BI
FCT --> BI
The standard layering. Staging models clean each source one-to-one; intermediate models hold reusable business logic; marts are the business-facing, often dimensional outputs that BI and metrics consume. The discipline is that each layer has one job — and downstream tools only ever touch marts.
- Staging — one model per source table, doing only the boring, mechanical work: rename columns to a convention, cast types, basic cleaning. No joins, no business logic. This is the single layer that knows what the raw source looks like, so a source change is contained here.
- Intermediate — the reusable building blocks: joins and business logic that more than one mart needs, factored out so it's defined once. This is where DRY lives.
- Marts — the business-facing models, usually dimensional (facts and dimensions), organized by business domain (finance, marketing). These are the products analysts and dashboards consume — and the only layer they should touch.
The reason this matters isn't tidiness for its own sake. Layering is what makes the project debuggable and changeable: a source schema change ripples only through staging; a metric change happens in one mart; an analyst never reaches past the marts into the plumbing. Skip the layers and you get the swamp I inherited — logic everywhere, changes that break things three hops away, no clean boundary anyone can reason about.
The semantic layer: one definition of revenue
The dueling-revenue-numbers problem has a specific cure: the semantic layer. Even with clean marts, if every BI tool and every analyst writes their own SUM(...) for "revenue" — one includes refunds, one filters test accounts, one uses a different date — you're back to inconsistent numbers, just one layer up. The semantic layer (dbt's, built on MetricFlow) lets you define a metric once, in version-controlled code, as a first-class object: revenue is this measure, over this entity, with these filters. Every consumer queries that definition, so the number is identical in the executive dashboard, the analyst's notebook, and the embedded report.
This is the same single-source-of-truth instinct that good BI semantic models chase, pushed down into the transformation layer so it's tool-agnostic. It's also the piece teams most often skip and most regret skipping — because the cost of inconsistent metrics isn't a bug ticket, it's eroded trust in the entire data platform. Once two executives have caught the dashboard contradicting itself, every future number is suspect.
Where analytics engineering hits its limits
I'm a believer, but the discipline has real ceilings, and pretending otherwise is how teams end up surprised:
| Limit | What it looks like in practice |
|---|---|
| SQL-shaped only | Transformations that aren't naturally SQL (complex Python, ML feature logic) fight the model. Python models help but it's not the sweet spot. |
| Model sprawl | "Everything is a model" becomes 800 models, half unused, with a lineage graph nobody can read. Governance debt creeps in quietly. |
| Mart explosion | A new mart per request instead of reusable intermediates — duplicated logic returns through the back door. |
| Warehouse compute cost | ELT means every transform runs on warehouse compute. Rebuild everything hourly and the bill teaches you about incremental models fast. |
| Modeling is still hard | dbt makes good modeling possible, not automatic. A clean toolchain over a bad data model is a well-tested swamp. |
The tooling can be immaculate and the data model still wrong. The seductive failure of analytics engineering is mistaking green tests and a tidy DAG for a good model. I've seen projects with perfect CI, full documentation, 100% test coverage — built on a dimensional model that didn't match how the business actually thinks, so every answer required heroic SQL and half the marts contradicted each other anyway. dbt enforces discipline; it does not supply judgment. The hard, unautomatable part is still deciding what the grain of a fact table is, what a customer is, and what revenue means — the dimensional modeling that no tool does for you. Test coverage measures whether the code does what you said; it says nothing about whether what you said was right.
What actually works
The practices that have paid off everywhere I've applied them:
- Hold the line on staging. One model per source, only renaming/casting/cleaning, no business logic. This single rule contains source changes and keeps the rest of the project sane.
- Factor shared logic into intermediates. The moment two marts need the same join or calculation, it belongs in an intermediate model — not copied. That's the whole DRY payoff.
- Test what matters, not everything. Primary keys unique and not-null, critical foreign keys, the handful of business rules that would be disasters if violated, source freshness. Testing every column to chase a coverage number just trains people to ignore the failures.
- Define each metric once in the semantic layer. The cost of skipping it is measured in lost trust, not bug tickets.
- Treat marts as products with owners. Naming conventions, documentation, an owner on the hook — and a data contract on the marts other teams depend on, so a refactor can't silently break them.
- Reach for incremental models before the rebuild-everything bill does it for you, and prune unused models periodically — sprawl is a cost, not a trophy.
What to carry away
Analytics engineering is software-engineering discipline applied to the transformation layer — version control, tests, modularity, code review, CI, documentation — on the work that turns raw data into trusted, modeled datasets. It created the analytics engineer role (analyst's SQL fluency, engineer's rigor), it was made necessary by the ELT-and-cloud-warehouse shift, and dbt is the tool that made it accessible. The structure is the staging → intermediate → marts layering, each layer with one job; the cure for inconsistent numbers is a semantic layer that defines every metric once.
The load-bearing caveat: the tool gives you discipline, not judgment. Green tests and a clean DAG over a wrong data model is a well-engineered swamp. The hard part is still the modeling — the grain, the entities, the definitions — which is why this pairs with dimensional modeling and warehouse design, and why the marts that matter deserve a data contract. Get the model right, then let the discipline keep it right.