Someone forwards you a benchmark. A graph-based retrieval system beats plain vector search by a large margin on some question set, with a chart. Your team wants to build the graph. And the question I always end up asking first is the one the chart never answers: what was the baseline?
Almost always it turns out to be pure dense retrieval with default chunking, top-k of five, no keyword channel and no reranker. That is not a baseline anyone should ship — it's the thing I spend most RAG engagements fixing, and hybrid search plus a reranker routinely beats it by a wide margin at a fraction of the cost of building a knowledge graph. Comparing a carefully-built graph pipeline against a deliberately naive vector one tells you nothing about whether you need a graph.
So this article is a method rather than a verdict. The comparison only means something once you split your questions by type, fix the baseline to something you'd actually deploy, and count the total cost of each pipeline rather than just its accuracy. Do that, and the answer stops being one number and becomes a routing decision — which, as it happens, is usually the right architecture too.
Why one number can't be the answer
Retrieval questions come in at least three shapes, and the shapes have genuinely different winners. This taxonomy is the single most useful thing I can give you here, because most disagreements about GraphRAG dissolve once both sides state which row they're arguing about.
| Question type | Example | What it needs | Usually wins |
|---|---|---|---|
| Local / factoid | "What's the notice period in the ACME contract?" | Find one passage, quote it | Hybrid + rerank. A graph adds latency and cost for nothing. |
| Multi-hop | "Which of our suppliers are owned by companies we've sanctioned?" | Join facts that never co-occur in one chunk | Graph, when the relationships are real and extractable |
| Global / thematic | "What are the recurring themes in last quarter's complaints?" | Summarize across the whole corpus, not retrieve from it | Graph community summaries — or a proper aggregation pipeline |
| Aggregate / numeric | "How many contracts renew in Q3?" | Counting | Neither. This is SQL wearing a question mark. |
The fourth row is included because it's the most common misdiagnosis I see. A question that requires counting or grouping is a text-to-SQL and semantic-layer problem, and no amount of retrieval sophistication — vector, hybrid or graph — will answer it reliably. If your evaluation set is full of aggregate questions, you are benchmarking two wrong tools against each other.
The published GraphRAG work is fairly explicit about this, incidentally: the case it makes is strongest for global sensemaking queries, where the answer requires the whole corpus rather than a passage from it. That's a real capability plain retrieval genuinely lacks. It is also, in most enterprise deployments I've seen, a minority of the actual question mix — which is why the headline comparisons and the production experience diverge so sharply.
Fixing the baseline
Before you compare anything to a graph, build the vector pipeline you would actually deploy. That means: hybrid retrieval (BM25 fused with dense vectors), a reranker over the fused candidates, structure-aware chunking rather than fixed character counts, and a top-k large enough to give the reranker something to work with. Every one of those is cheap, well-understood, and standard practice.
In my experience this alone closes most of the gap that naive-baseline benchmarks show, particularly on local and multi-hop questions where the relevant facts happen to live in retrievable proximity. It also changes what you learn from the comparison: instead of "graphs are better," you find out which specific question types still fail after you've done the ordinary work — and those are the ones worth building a graph for.
Two more baseline honesty checks. Give both pipelines the same generation model and prompt, or you're measuring the prompt. And be explicit about top-k and context budget — a graph pipeline that feeds the model 8,000 tokens of community summaries is not comparable to a vector pipeline capped at 2,000, and the difference in the answer may be entirely context volume.
graph TB
Q[("Question set from
REAL production traffic")] --> CLASS{"Classify by type"}
CLASS -->|"local / factoid"| L["Local set"]
CLASS -->|"multi-hop"| M["Multi-hop set"]
CLASS -->|"global / thematic"| G["Global set"]
CLASS -->|"aggregate"| A["Aggregate set
→ route to SQL, exclude"]
L --> RUN["Run BOTH pipelines
same model, same prompt,
same context budget"]
M --> RUN
G --> RUN
RUN --> B1["Baseline: hybrid + rerank
structure-aware chunks"]
RUN --> B2["Graph: entity + relation
extraction, traversal,
community summaries"]
B1 --> SCORE["Score per type:
answer correctness · groundedness
· recall of required facts"]
B2 --> SCORE
SCORE --> COST["Add the other columns:
build cost · update cost
· query latency · $ / question"]
COST --> DEC{"Per type, is the
graph's win worth its cost?"}
The shape of an honest comparison. Two details do most of the work: questions come from real traffic, classified by type, rather than from a benchmark set chosen to showcase multi-hop reasoning — and the cost columns sit on the same table as the accuracy columns, because a two-point gain that costs a full-time engineer to maintain is not a win you can bank.
Measuring the right things
Score per question type, never in aggregate — an average across a mix you chose is a number you can tune by changing the mix. Within each type:
- Required-fact recall. For each question, list the facts a correct answer must contain, and measure how many the retrieval surfaced. This is the metric that isolates retrieval from generation, and it's the one that tells you why a pipeline lost.
- Answer correctness against a reference or rubric, and groundedness — is the answer supported by what was retrieved? Graph pipelines that summarize aggressively can score well on correctness while being harder to ground, because the summary is now the source rather than the document.
- Citation traceability. Can a human get from the answer back to a document? This matters more than benchmarks admit — an answer synthesized from community summaries may be right and unverifiable, which in a regulated setting is not an acceptable trade.
Then the columns that decide real projects, which almost no published comparison includes:
| Cost dimension | Hybrid + rerank | Graph pipeline |
|---|---|---|
| Build | Chunk + embed once | LLM-driven entity and relation extraction over the whole corpus — often the dominant one-off cost |
| Update | Re-embed changed documents | Re-extract, reconcile entities, rebuild affected communities |
| Query latency | One retrieval + one rerank | Traversal, possibly multiple LLM calls |
| Failure mode | Misses the passage | Extracts the wrong relationship and then reasons confidently over it |
| Who maintains it | Existing data pipeline | Someone who understands the ontology — a role, not a task |
That last row is the one I'd put in front of a sponsor. A knowledge graph is not a component you add; it's a modelling commitment that needs an owner, because entity resolution decisions and ontology drift are ongoing work. The update column matters just as much for a corpus that changes weekly — re-extraction is not free, and a stale graph is worse than a stale index because its errors are structural rather than merely absent.
The failure mode that makes graphs risky in production: confident reasoning over a bad edge. When hybrid search fails, it returns a passage that doesn't answer the question, and the model usually says so. When LLM-driven extraction produces a wrong relationship — merging two entities with similar names, inferring an ownership link from adjacent sentences — the graph now contains a fact that isn't true, and the pipeline reasons over it with full confidence and a clean citation trail. That's harder to detect, harder to debug, and considerably more dangerous in a domain where someone acts on the answer. So if you build one: measure extraction precision on a hand-checked sample before you measure anything downstream, and treat entity resolution as the highest-risk step rather than a preprocessing detail. A graph with 85% edge precision can look excellent on an eval set and be unusable for a compliance question.
The architecture the comparison usually points to
Run this exercise honestly and you rarely get "build a graph" or "don't." You get a distribution — some question types where hybrid wins outright, some where a graph wins clearly, and some where the difference doesn't justify the cost. Which means the sensible architecture is a router: classify the incoming question, send local and factoid queries to hybrid retrieval, send multi-hop and global ones to whatever handles them best, and send aggregate ones to SQL where they belonged all along.
Two things make this practical. The classifier is a cheap, small-model call or even a set of heuristics, and it's easy to evaluate on its own. And it lets you build incrementally: ship hybrid retrieval, measure which question types fail, and add the graph only for the types that need it — with the eval set already in place to prove it helped.
There's also a middle option worth knowing about. For a bounded corpus you can rebuild deterministically, a build-time graph over structure you already have — links, citations, tags, foreign keys — gives you a large share of multi-hop capability with none of the extraction risk, because the edges are facts rather than inferences. It's a much smaller commitment than LLM-extracted knowledge graphs and it's where I'd start whenever the corpus permits it, a theme I've explored around knowledge graphs and the semantic web.
What to carry away
Treat any GraphRAG-versus-vector comparison as unfinished until you know the baseline, and assume it was naive unless told otherwise. Fix the baseline first — hybrid retrieval, a reranker, structure-aware chunking, a fair top-k — because that closes most of the gap on most question types at a fraction of the cost.
Split your questions by type before scoring: local and factoid (hybrid wins), multi-hop (graph wins when relationships are real and extractable), global and thematic (graph summaries, or an aggregation pipeline), aggregate (neither — that's SQL). Score required-fact recall per type, not one average across a mix you selected. Put build cost, update cost, query latency and maintenance ownership in the same table as accuracy, because those are what the decision actually turns on.
If you do build a graph, measure extraction precision on a hand-checked sample first — the dangerous failure isn't a miss, it's confident reasoning over an edge that shouldn't exist. And take the routing architecture the evidence points to rather than the single-pipeline answer the benchmarks imply: classify the question, and send it to the retrieval strategy that has earned that type.