Building RAG on Azure: AI Search, Cosmos DB, pgvector, or Qdrant?

The Azure RAG system demoed beautifully. Ask it a question, get a grounded answer with citations, everyone nods. Then it met real users and real documents, and the answers went soft — confidently retrieving the wrong paragraph, missing the obvious one, citing a document that didn't contain the claim. The instinct in the room was to reach for a bigger model. The problem was never the model. It was retrieval, and on Azure the retrieval decision that matters most is which store you put your chunks in.

This is a build guide for retrieval-augmented generation on Azure and Fabric, written from the position I actually hold: the model is a rented commodity, and the quality of a RAG system is decided almost entirely upstream of it — in chunking, in the retriever, and in the reranker. I'll walk the minimal pipeline, make the store decision between Azure AI Search, Cosmos DB, pgvector on Azure PostgreSQL, and Qdrant — with an honest default — cover the handful of practices that moved quality more than anything else, and share the lessons that cost me. It's a companion to the versions I've written for GCP, Snowflake, and AWS; the shape is the same, the components are Azure's, and the trade-offs are different enough to matter. Everything here is as of early 2026.

The minimal RAG pipeline on Azure

A production RAG system on Azure needs exactly five moving parts: a source of documents, a chunker, an embedding model, a retriever backed by a vector store, and a generation call — plus an evaluation harness you build before you scale. Everything else people bolt on early is usually premature. You do not need a graph database, a feature store, and three vector engines to answer questions over your documents; you need the smallest pipeline that retrieves the right chunk, and the discipline to measure whether it does.

graph LR
  SRC[("Documents
OneLake / Blob / SharePoint")] --> CH["Chunk
structure-aware + overlap"]
  CH --> EMB["Embed
Azure OpenAI text-embedding-3"]
  EMB --> IDX[("Vector store + retriever")]
  Q["User question"] --> RET["Retrieve
hybrid + rerank"]
  IDX --> RET
  RET --> GEN["Generate
Azure OpenAI GPT-4o"]
  GEN --> ANS["Grounded answer + citations"]
  EVAL["Evaluation harness"] -.->|"groundedness, recall"| RET

A word on Fabric specifically, because the question I get is "can I just do RAG in Fabric?" OneLake is the right home for the documents and the ingestion — a Fabric pipeline or notebook to land, clean, and chunk them, with the lakehouse as the durable source of truth. Fabric's Eventhouse/KQL can compute vector similarity, and it's genuinely useful when your data is already telemetry in KQL. But for document RAG, the retrieval engine that gives you the best answers with the least work is Azure AI Search, and the clean architecture is Fabric-for-data, AI-Search-for-retrieval. Don't force the vector index into Fabric because the data starts there.

Which retrieval store? The decision that matters

Azure AI Search is the right default for document RAG on Azure, and the other three win in specific, nameable situations. The reason AI Search is the default isn't that it's the fastest vector index — it's that retrieval quality comes from hybrid search plus semantic reranking, and AI Search does both natively, along with the enterprise plumbing (security trimming, managed indexing) that a real deployment needs. The alternatives are about keeping vectors next to data you already run.

Azure AI Search — the default, and why

Azure AI Search earns the default because of three GA capabilities that together decide quality. Integrated vectorization lets the indexer chunk and embed your documents on the way in — you point it at a data source and a skillset, and it calls Azure OpenAI to vectorize without you writing an ingestion job. Hybrid search runs a BM25 keyword query and a vector query together and fuses them with reciprocal rank fusion, which catches both the semantic match and the exact-term match that pure vectors miss. And the semantic ranker re-scores the top results with a language model, which in my experience is the single biggest quality lever on the platform. On top of that it does security trimming — document-level access filters — which is a hard requirement the moment your corpus has permissions, and a pain to build yourself elsewhere.

There's a newer agentic retrieval capability — LLM-assisted query planning that decomposes a question and searches in parallel — but as of early 2026 it's in preview. I've shipped the classic hybrid-plus-semantic-ranker pattern to production and kept an eye on agentic retrieval for when it's GA; for anything that needs to be dependable now, the GA path is the one to build on.

Cosmos DB for NoSQL — vectors next to your operational data

Cosmos DB added native vector search with a DiskANN index (Microsoft Research's algorithm), and it's the right call when the data you're grounding on already lives in Cosmos — user records, product catalogues, session state. Keeping the vectors in the same container as the operational document means no synchronisation pipeline and a transactional read of both. DiskANN gives low latency and, importantly on Cosmos, low RU cost per query, with support for high-dimension vectors. The trade-off is that Cosmos is a database, not a search engine — you get vector similarity, but hybrid fusion and semantic reranking are yours to assemble, and RU consumption is the number to watch before it surprises you on the bill.

pgvector on Azure PostgreSQL — one datastore

If your application already runs on Azure Database for PostgreSQL Flexible Server, pgvector keeps the embeddings in the same database as everything else — transactional upserts, one backup, one thing to operate, and SQL to combine vector distance with your existing filters. Azure added a DiskANN index for Postgres (GA in 2025) that materially outperforms pgvector's stock HNSW on memory and speed for large sets, which softens pgvector's usual scaling ceiling. The honest limit is the same as pgvector anywhere: single-node, and you build hybrid and reranking on top rather than calling them. For a team already on Postgres, that's often a fair trade for not running a second system — the same argument I made in the vector databases radar.

Qdrant — when you need the dedicated engine

Qdrant is the pick when you genuinely need a purpose-built vector engine's features — advanced payload filtering that stays fast under selective predicates, aggressive quantization to fit large sets in memory, fine control over the index — and you're willing to run it on AKS or via its managed marketplace offering. It's an excellent engine. On Azure specifically, the bar it has to clear is "worth operating a dedicated system that AI Search's managed service would otherwise handle," and for most document-RAG workloads that bar is high. Reach for it when filtering or scale is the constraint that AI Search can't meet economically, not by default.

What actually moved retrieval quality

The store decision sets your ceiling; these practices are how you reach it. In order of how much they mattered on real projects.

Hybrid + semantic reranking, not pure vector

The biggest, cheapest win: do not ship pure vector search. Vector-only retrieval misses exact terms — a product code, a proper noun, a specific error string — that a keyword query catches instantly, and it has no notion of which of the top-20 semantic matches actually answers the question. Hybrid search fuses keyword and vector; the semantic ranker then reorders the fused top results by relevance to the query. On AI Search you get both as configuration. On the other stores you build the equivalent — keyword index alongside the vectors, a fusion step, and ideally a reranker. Every time I've measured it, hybrid-plus-rerank beat pure vector by a margin that dwarfed the difference between embedding models.

Chunking is a retrieval decision, not a preprocessing chore

Chunk too large and the embedding blurs across several ideas so nothing matches cleanly; chunk too small and you retrieve a sentence with no context for the model to ground on. The pattern that worked: structure-aware chunks — split on the document's own boundaries (headings, sections) rather than a blind character count — with modest overlap so a fact that straddles a boundary survives, and metadata on every chunk (source, section, date, access tags) so you can filter and cite. Integrated vectorization gives you a reasonable default splitter; custom chunking earns its keep when your documents have real structure worth respecting.

Security trimming, from day one

The moment your corpus has permissions — and enterprise corpora always do — retrieval has to respect them, or the RAG system becomes a beautiful data-exfiltration tool that cheerfully quotes a document the user can't open. Filter at the retriever with the user's identity, not after generation. AI Search does document-level security trimming natively; on the database options you enforce it with a filter predicate on every query. Designing this in from the start is a governance decision, and retrofitting it is the kind of rework that a little upfront thought avoids — the same lesson as tokenizing PII before it spreads.

Evaluate before you scale

You cannot improve what you don't measure, and "it looked good in the demo" is not a measurement. Before scaling, build an evaluation set of real questions with known-good answers and score two things: retrieval recall (did the right chunk come back at all?) and groundedness (is the answer supported by what was retrieved, or is the model filling gaps?). Azure AI Foundry ships evaluation tooling for exactly this, but a spreadsheet of questions and a scoring script beats no evaluation by a mile. Retrieval recall in particular is the diagnostic that tells you whether your problem is the store, the chunking, or the model — usually it's not the model.

Here's the classic pattern in code — integrated vectorization means you send text, not a vector, and Search embeds the query for you, runs hybrid, and applies the semantic ranker:

# Azure AI Search: hybrid (keyword + vector) query with semantic reranking.
# Integrated vectorization embeds the query text server-side, so the client
# never touches an embedding model at query time.
from azure.search.documents import SearchClient
from azure.search.documents.models import VectorizableTextQuery
from azure.identity import DefaultAzureCredential

search = SearchClient(
    endpoint="https://my-search.search.windows.net",
    index_name="docs",
    credential=DefaultAzureCredential(),   # managed identity, not a key in code
)

question = "What is our refund window for enterprise contracts?"

results = search.search(
    search_text=question,                              # BM25 keyword side
    vector_queries=[VectorizableTextQuery(             # vector side, embedded server-side
        text=question, k_nearest_neighbors=50, fields="content_vector",
    )],
    query_type="semantic",                             # turn on the reranker
    semantic_configuration_name="default",
    filter="access_group eq 'enterprise'",             # security trimming, at the retriever
    top=8,
)

for r in results:
    # @search.reranker_score is the semantic ranker's judgment — sort/threshold on it
    print(r["@search.reranker_score"], r["title"], r["content"][:160])

A reference architecture

The shape I'd build for a document-RAG system on Azure, keeping the retriever behind an interface so the store stays a swappable decision rather than a rewrite:

graph TD
  subgraph Data["Data layer — Microsoft Fabric"]
    OL[("OneLake / Lakehouse")]
    PIPE["Pipeline / notebook
land + clean + chunk"]
    OL --> PIPE
  end
  subgraph Retrieval["Retrieval layer"]
    AIS[("Azure AI Search
integrated vectorization
hybrid + semantic ranker")]
  end
  subgraph App["Application layer"]
    ORCH["Orchestrator
Azure AI Foundry / your app"]
    LLM["Azure OpenAI
GPT-4o"]
  end
  PIPE --> AIS
  AOAI["Azure OpenAI
text-embedding-3"] -.->|"embeds during indexing"| AIS
  USER["User"] --> ORCH
  ORCH -->|"hybrid query + identity filter"| AIS
  AIS -->|"top-k reranked chunks"| ORCH
  ORCH --> LLM
  LLM --> USER
  ENTRA["Microsoft Entra ID"] -.->|"identity for security trimming"| AIS

What to carry away

On Azure, the model is the commodity and retrieval is the product. Default to Azure AI Search because hybrid search plus the semantic ranker plus native security trimming give you the best answers with the least code, and because integrated vectorization removes the ingestion job you'd otherwise write. Move to Cosmos DB or pgvector when the point is to keep vectors next to data you already run, and to Qdrant when you genuinely need a dedicated engine's filtering or scale and will operate it. Let Microsoft Fabric be the data and ingestion layer it's good at, and don't force the vector index into it.

Then spend your effort where quality actually lives: ship hybrid with reranking, not pure vector; treat chunking as a retrieval decision; trim by security from day one; and measure retrieval recall and groundedness before you scale, because the failure you don't measure is the one your users find first. Build the smallest pipeline that retrieves the right chunk, and add nothing until the evaluation says you need it. If you're still deciding whether RAG is even the right tool, that's fine-tuning vs RAG vs prompting; the store decision in depth is the vector databases radar; and for the platform trade-off underneath, Fabric vs Databricks.