# GraphRAG vs Hybrid Search: How to Actually Compare Them

*Published 2025-11-11 · Dmitry Shirokov · shirokoff.ca/blog/graphrag-vs-hybrid-search-evaluation*

Someone forwards a benchmark: graph retrieval beats vector search by a large margin, with a chart. The question the chart never answers is *what was the baseline?* Almost always: pure dense retrieval, default chunking, top-k of five, no keyword channel, no reranker — not a baseline anyone should ship, and the thing I spend most RAG engagements fixing. **The comparison only means something once you split questions by type, fix the baseline to something you'd deploy, and count total cost rather than accuracy alone.**

## Why one number can't be the answer

| Question type | Example | Needs | Usually wins |
|---|---|---|---|
| **Local / factoid** | "Notice period in the ACME contract?" | Find one passage | **Hybrid + rerank** — a graph adds cost for nothing |
| **Multi-hop** | "Which suppliers are owned by sanctioned companies?" | Join facts that never co-occur | **Graph**, when relationships are real and extractable |
| **Global / thematic** | "Recurring themes in last quarter's complaints?" | Summarize *across* the corpus | **Graph community summaries**, or an aggregation pipeline |
| **Aggregate / numeric** | "How many contracts renew in Q3?" | Counting | **Neither** — this is SQL wearing a question mark |

The fourth row is the most common misdiagnosis: an eval set full of aggregate questions benchmarks two wrong tools against each other. The published GraphRAG case is strongest for **global sensemaking**, which is a real capability plain retrieval lacks — and a minority of the question mix in most enterprise deployments, which is why headline comparisons and production experience diverge.

## Fix the baseline

Hybrid retrieval (BM25 fused with dense), a reranker over the fused candidates, structure-aware chunking, a top-k big enough for the reranker to work with. All cheap and standard. This closes most of the gap naive-baseline benchmarks show, and changes what you learn: instead of "graphs are better," you find which question types still fail after the ordinary work — and those are the ones worth a graph. Also: **same generation model and prompt** for both, and an explicit **context budget** — a graph pipeline feeding 8,000 tokens of summaries isn't comparable to a vector pipeline capped at 2,000.

## Measure per type, and include cost

Score **required-fact recall** (list the facts a correct answer must contain; measure how many retrieval surfaced) — it isolates retrieval from generation and tells you *why* a pipeline lost. Plus correctness, groundedness, and **citation traceability**: an answer synthesized from community summaries can be right and unverifiable, which is not an acceptable trade in a regulated setting.

| Cost | Hybrid + rerank | Graph |
|---|---|---|
| Build | Chunk + embed once | LLM extraction over the whole corpus — often dominant |
| Update | Re-embed changed docs | Re-extract, reconcile entities, rebuild communities |
| Query latency | One retrieval + rerank | Traversal, possibly multiple LLM calls |
| Failure mode | Misses the passage | **Extracts a wrong relationship and reasons confidently over it** |
| Maintainer | Existing data pipeline | Someone who owns the ontology — a role, not a task |

## The risky failure

When hybrid search fails it returns an unhelpful passage and the model usually says so. When extraction produces a wrong edge — merged entities, an ownership link inferred from adjacent sentences — the graph contains an untrue fact and the pipeline reasons over it confidently, with a clean citation trail. Measure **extraction precision on a hand-checked sample** before anything downstream; 85% edge precision can look excellent on an eval set and be unusable for a compliance question.

## Where the evidence usually points

Not "build a graph" or "don't" — a distribution. Which makes the sensible architecture a **router**: classify the question, send local/factoid to hybrid, multi-hop and global to whatever handles them, aggregate to SQL. The classifier is a cheap small-model call, independently evaluable, and it lets you ship hybrid first and add a graph only for the types that need it, with the eval set already in place to prove it helped. Middle option: for a bounded corpus, a **build-time graph over structure you already have** (links, citations, tags, foreign keys) gives much of the multi-hop capability with none of the extraction risk, because the edges are facts rather than inferences.

## Carry-away

Assume the baseline was naive unless told otherwise; fix it first. Split questions by type and score required-fact recall per type. Put build cost, update cost, latency and maintenance ownership in the same table as accuracy. If you build a graph, measure extraction precision first — the dangerous failure isn't a miss, it's confidence over an edge that shouldn't exist. Then take the routing architecture the evidence points to.

*Related: [GraphRAG](https://shirokoff.ca/blog/graphrag) · [Knowledge graphs and the semantic web](https://shirokoff.ca/blog/knowledge-graphs-semantic-web) · [RAG on GCP](https://shirokoff.ca/blog/rag-on-gcp) · [Vector databases](https://shirokoff.ca/blog/vector-databases)*